akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Libtorch C++ batch prediction in parallel

marchss: <p>Hi, the pytorch model can be converted into C++ and  forward() prediction is also successful. However, I need to speed up the forward prediction by processing batches of inputs in parallel.</p>
<p>Specifically, say batch_size = 32, inputs contains num_samples = 64, then we have num_batches = batch_size / num_samples = 2, A and B. If using a for-loop to iterate and do forward(A), forward(B) sequentially, it will be slow.  To speed up, how can we do forward(A) and forward(B) in parallel?</p>
<p>Is below enough?</p>
<pre><code class="lang-auto">#pragma omp parallel
#pragma omp for
    for(i = 0; i &lt; num_batches; i++)
    {
        //do model-&gt;forward(batch[i]);
    }
</code></pre>
<p>Does anyone have a minimal C++ example to do that?</p>
How does backward work with JIT module?

eee4447b4b8a2cbdfbbd: <p>I’ve read README.md about JIT at (<a href="https://github.com/WyldeCat/pytorch/blob/master/torch/csrc/jit/README.md" rel="nofollow noopener">https://github.com/WyldeCat/pytorch/blob/master/torch/csrc/jit/README.md</a>).</p>
<p>There is a great explanation about how JIT handles a module’s forward method.</p>
<p>But I can’t find any explanation about how backward pass works.</p>
<p>So, I want to ask whether JIT involves in backward pass or not and is there any way to print backward pass’s graph like JIT module’s forward graph.</p>
<p>Thanks in advanced.</p>
Sparsifying neural networks

shreyd: <p>I’m seeing a lot of literature on the sparsification of neural networks, but I’m confused the benefit of said methods. Once we sparsify a network (say through various pruning methods), how do we actually save space in the final model? I see a sparse flag in the Linear layers, so do you set that before/after pruning?</p>
<p>Also related to the questions above, what does it mean to “prune” a network? Do you zero out the weights (but they still exist)? Not count them in the final parameter count?</p>

JuanFMontesinos: <p>The idea in behind is that neural networks are (in general) overparametrized. In theory, the more parameters it a DNN has, the easier it converges.</p>
<p>Idea in behind is that once a DNN has been trained you can get rid off these non useful parameters. I don’t know how it is implemented in practice. I guess it will be some kind of sparse tensor.</p>
Get filtered model parameters

tsauri: <p>Say I have a model,</p>
<p>How do I get model.parameters() but filter out params with has certain name (e.g. “conv1”, “conv2”)</p>
<p>This is because I want to make two optimizers, where both optimize different params</p>
<p>Thank you</p>

JuanFMontesinos: <p>hi, there exist another method called named_parameters:</p>
<pre><code class="lang-auto">for name,param in model.named_parameters():
    do your stuff
</code></pre>
Model summary and reshaping the output of one layer to feed to next layer

Kaushal_Aggarwal: <p>how to do I print model summary like what we do in keras model.summary().<br>
Also I am using conv1d and I need to use the output of this to conv2d for which I need to add to reshape the output of conv1d. How to do that?</p>

Balamurali_M: <p>print (model) itself will give a summary of it’s properly represented in network class.</p>
<p>Torch. Reshape or view can be used to convert 1d output to 2d output.</p>

sksq96: <aside class="quote quote-modified" data-post="7" data-topic="9879">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sksq96/48/13282_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-print-a-model-after-load-it/9879/7">How to print a model after load it?</a> 
  </div>
  <blockquote>
    Yes, you can get exact Keras representation, using this <a href="https://github.com/sksq96/pytorch-summary" rel="noopener nofollow ugc">code</a>. 
Example for VGG16
from torchvision import models
from summary import summary

vgg = models.vgg16()
summary(vgg, (3, 224, 224))

----------------------------------------------------------------
        Layer (type)               Output Shpae         Param #
================================================================
            Conv2d-1         [-1, 64, 224, 224]            1792
              ReLU-2         [-1, 64, 224, 224]    …
  </blockquote>
</aside>
How to get the backwad function of Conv2d?

Wenlong_Shi: <pre><code class="lang-auto">output = conv2d_forward(input, weight)
grad_input, grad_weight = conv2d_backward(grad_output, input, weight)
</code></pre>
<p><code>conv2d_forward</code> can be found in torch.nn.functional.conv2d, where can i get <code>conv2d_backward</code> function?</p>
360 hr Librispeech training for neural network

pytorch_user12: <p>Hi, I want to train librispeech which contains variable length of samples.<br>
How to use Dataloader for this ( should I Cut it into framewise and then train) or is their any methods for loading different sample size.<br>
If I follow to cut it into frames wise,  its a huge number files which it able to handle properly by the GPU.<br>
Is their any method to write a customized way to train the neural network.</p>
[solved] Expected tensor for argument #1 'indices' to have scalar type Long; but got CPUDoubleTensor instead (while checking arguments for embedding)

dendisuhubdy: <p>A little bit of context</p>
<pre><code class="lang-auto">        dlg_turnwise_embeddings = np.random.random((batch_size, 20, 256))
        dlg_turnwise_embeddings = torch.from_numpy(dlg_turnwise_embeddings)

        if legacy:
            hidden = rnn_model.init_hidden(batch_size)
            # words_emb: batch_size x nef x seq_len
            # sent_emb: batch_size x nef
            words_emb, sent_emb = rnn_model(captions, cap_lens, hidden)
        else:
            words_emb = word_reducer(data_caption_word_embeddings)
            print(data_caption_word_embeddings)
            sent_emb = sent_reducer(data_caption_embeddings)
            # the input to the model is here
            # data caption word embedding
            # this would be changed to data_dialogue_word_embedding
            # words_emb  torch.Size([48, 256, 14])
            # we want the same size for the word and sentence embedding
            # sent_emb  torch.Size([48, 256])

            # TODO: write an LSTM model that outputs the following
            # with size ((batch_size, 256))
            hidden = dlg_encoder.init_hidden(batch_size)
            dlg_entire_embeddings = dlg_encoder(dlg_turnwise_embeddings, 
                                                dlg_turnwise_embeddings.shape,
                                                hidden)
</code></pre>
<p>Seems to happen when I do <code>dlg_turnwise_embeddings = torch.from_numpy(dlg_turnwise_embeddings)</code>? Is it as simple as casting it to Long from Double values inside the tensor?</p>

dendisuhubdy: <p>Nevermind figured it out</p>
<pre><code class="lang-auto"> |  109         dlg_turnwise_embeddings = np.random.random((batch_size, 20, 256))                                                          
 |&gt;&gt;110         dlg_turnwise_embeddings = np.asarray(dlg_turnwise_embeddings, dtype=int)                                                   
 |+ 111         dlg_turnwise_embeddings = torch.from_numpy(dlg_turnwise_embeddings)                                                        
 |+ 112         if cfg.CUDA:                                                                                                               
 |+ 113             dlg_turnwise_embeddings = dlg_turnwise_embeddings.cuda() 
</code></pre>

marchss: <p><a class="mention" href="/u/dendisuhubdy">@dendisuhubdy</a> So, the final solution is to convert the dtype to int?</p>
<aside class="quote no-group" data-username="dendisuhubdy" data-post="2" data-topic="16979">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dendisuhubdy/48/844_2.png" class="avatar"> dendisuhubdy:</div>
<blockquote>
<p>dlg_turnwise_embeddings</p>
</blockquote>
</aside>
Multiple loss function

Rojin: <p>I am building an autoencoder, and I would like to take the latent layer for a regression task (with 2 hidden layers and one output layer).</p>
<p>This means that I have two loss functions, one for the AE and one for the regression. I have a few questions:</p>
<ol>
<li>Do you suggest adding up both loss values and backprop?</li>
<li>If I want to backprop each model with respect to its own loss value, how should I implement that in pytorch? should I call loss1.backward() and loss2.backward() then opt.step() ?</li>
</ol>

JuanFMontesinos: <p>Sum them up, the other way around is not feasible at all as you are trying to optimize 2 different latent spaces</p>

Rojin: <p>Dear <a class="mention" href="/u/juanfmontesinos">@JuanFMontesinos</a></p>
<p>I am optimizing the followings:</p>
<ol>
<li>The VAE</li>
<li>The regression</li>
</ol>
<p>And the input to the regression is the latent space of the VAE.</p>
<p>I would like to use the regression loss to change the weight of the regression portion of the model, and the VAE loss to change the VAE  weights</p>

JuanFMontesinos: <p>Then, backpropagate regression loss on those layers. You just have to detach regression loss not to backprop it farther than it is necessary. 2 backwards should be ok</p>

Rojin: <p>Yes, I wanna know how I can do that in Pytorch</p>

JuanFMontesinos: <pre><code class="lang-auto">latent_space, output = VAE(input)
classification = classifier(latent_space.deatch())
</code></pre>
<p>Basically whatever you pass to that fully connected layer, detach() it, it breaks backprop graph.</p>
<p>If there were an issue while backpropagating, you should see a warning saying certain layer’s graph has already been used.</p>
ReduceLROnPlateau parent class is not _LRScheduler

sahni: <p>I was wondering why ReduceLROnPlateau does not have _LRScheduler as parent class. If I want to check if a variable is a learning rate scheduler object type, I could have used:</p>
<pre><code class="lang-auto">isinstance(scheduler, optim.lr_scheduler._LRScheduler)
</code></pre>
<p>Above covers all learning rate schedulers defined in lr_scheduler.py except ReduceLROnPlateau because it does not inherit from _LRScheduler. So instead I need to use:</p>
<pre><code class="lang-auto">isinstance(scheduler, optim.lr_scheduler._LRScheduler) or isinstance(scheduler, optim.lr_scheduler.ReduceLROnPlateau)
</code></pre>
<p>Shouldn’t ReduceLROnPlateau inherit from _LRScheduler as well? Is this just an oversight?</p>
<p>Context:<br>
In general for saving a checkpoint, if I also want to save scheduler state dict, then I would like to verify the params first.</p>
Using softmax causes out of memory error for CUDA

tnguyen: <p>I am writing the Transformer that Google proposes through their paper “Attention is All You Need”. The problem that I’m running into is that, when I apply a softmax to my final logits before outputting them, I run into the error <code>RuntimeError: CUDA out of memory. Tried to allocate 1.38 GiB (GPU 0; 11.91 GiB total capacity; 8.91 GiB already allocated; 841.06 MiB free; 1.58 GiB cached)</code>. The thing is that, I keep track of my tensors by printing out (.element_size() * .nelement()). I see that with and without softmax, my output tensors are around 1.5 GB in size, but when I apply softmax to the outputs, by calling torch.nn.functional.softmax, I get the error above and I’m not sure why. I’m getting this error when I call loss.backward() after calculating loss, but it only happens when I use softmax on the outputs.</p>
<p>The code is really long so I suppose it’s not very pleasant to read through so much code. Does anyone know what could be the problem or any suggestions to try?</p>
Why does the translation tutorial not use batches?

pinocchio: <p>I was going through the <a href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html#sphx-glr-intermediate-seq2seq-translation-tutorial-py" rel="nofollow noopener">TRANSLATION WITH A SEQUENCE TO SEQUENCE NETWORK AND ATTENTION</a> and was wondering why this tutorial did process the data in batches? Is it because dealing with variable lengths is to much for one tutorial?</p>
Can I manually edit/simplify the computational graph generated by autograd?

NicolaDV: <p>Automatic differentiation is really helpful - but sometimes manual differentiation can be better.</p>
<p>I heavily optimized my network structure use inplace operations, and I got a 40x speedup in the forward phase. The backward phase is still very slow, and now it takes about 90% of the time required by one step.</p>
<p>I’ve looked at the graph generated by .backward(), and it’s sprawling!!! Many of the sub-graph are identical and differ only in the input vector (the same sub-network is referred to multiple time). I think could get a significant speedup by reducing all these identical graphs to just one, and by combining the input gradients.</p>
<p>The question is, how can I do that?</p>
How to implement combining global and mid-level features in CNN?

bit_comedian: <p>Hey, there!</p>
<p>I have some problem with my CNN model.</p>
<p>It takes image at input, then pushes image forward to 2 submodels -  midlevel feature extractor and global feature extractor. Some image for understanding provided below: The output shape of midlevel feature extractor is (64,28,28), the size of global feature extractor verctor is 64.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/8/844cae8c8c0ea039e90a6b74e4b14dff48e68799.png" data-download-href="https://discuss.pytorch.org/uploads/default/844cae8c8c0ea039e90a6b74e4b14dff48e68799" title="untitled-3-2"><img src="https://discuss.pytorch.org/uploads/default/original/2X/8/844cae8c8c0ea039e90a6b74e4b14dff48e68799.png" alt="untitled-3-2" data-base62-sha1="iSnjBloWXq5u4G9p8F2WdWJHj3r" width="675" height="500" data-dominant-color="E1FAE1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">untitled-3-2</span><span class="informations">796×589 3.12 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I want to merge these ouputs in channel dimension per pixel, so the total output would be the size of (128, 28, 28). Another image is provided below.</p>
<p>Can someone advice what it the best way to do it in pytorch?</p>
<p>I tried to use expand method:</p>
<pre><code class="lang-auto">    def forward(self, x):
        x1 = self.midlevel_feature_extractor(x) #midlevel features
        x2 = self.global_feature_extractor(x) #global features

        #expanding
        x2 = x2.expand((x1.size()[2], x1.size()[3], x1.size()[0], x1.size()[1])).permute((2,3,0,1))
        
        #concat
        concatenated_features = torch.cat((x1, x2), dim=1)
        ...
</code></pre>
<p>But it result to error during training (I think problem is in wrong concatenating)</p>
<p>So, any ideas, guys?</p>

bit_comedian: <p>I figured out, how to do this.</p>
<p>just add at initialization of model</p>
<pre><code class="lang-auto">self.ups = nn.Upsample(28)
</code></pre>
<p>and change forward as</p>
<pre><code class="lang-auto">    def forward(self, x):
        x1 = self.midlevel_feature_extractor(x) #midlevel features
        x2 = self.global_feature_extractor(x) #global features
        x2 = x2.view(x2.size()[0], x2.size()[1], 1, 1)
        x2 = self.ups(x2)

        #concat
        concatenated_features = torch.cat((x1, x2), dim=1)
        ...
</code></pre>
How can I train 2 submodules in parallel with 2 GPUs - one per GPU?

nathan: <p>Hi, I have loaded my 2 submodules in 2 GPUs - one per GPU as below</p>
<p>submodule1.cuda(0)<br>
submodule2.cuda(1)</p>
<p>and in the forward pass</p>
<p>x1’ = submodule1(x1)<br>
x2’ = submodule2(x2)</p>
<p>In the above case, GPU1 is idle while GPU0 is processing. Is it possible to run the above 2 in parallel and not one after the other?</p>

Stone: <p><a class="mention" href="/u/nathan">@nathan</a> you may need to invoke two threads each for managing one GPU, because although the CUDA calls are async by default, you might introduce some python-level sync point in your module.<br>
Could you post up a sample code of your submodule?</p>

nathan: <p>Thanks <a class="mention" href="/u/stone">@Stone</a>. I created python multiprocessing Pool to execute them parallely.</p>
<p>But the problem was time-taken(execute them using Pool) &gt; time-taken(execute them sequentially)!</p>
<p>Probably due to the bookeeping involved with Pool. Any thoughts?</p>

Milky: <p>Hey nathan, I was thinking of doing the same, I have 2 models I wanted to optimize in parallel using 2 GPUs however I see you said it takes longer, was this a bug or is it really the case? If so I will not bother with it, unless it was easy to implement and try?</p>
Using streams to seperate conv branches on same gpu

mellorjc: <p>Hi,</p>
<p>I was trying to use streams to speed up calling multiple Conv2d modules on the same GPU.</p>
<p>My code is below.</p>
<p>It doesn’t appear to run any quicker . There was a previous question asked last year about using streams. There was a suggestion that all ops should be run on non-default streams. I tried to accomplish this but my attempts don’t seem to have helped.</p>
<p>Is there any obvious problem with my approach?</p>
<p>Thanks</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn


class ParallelDilatedConv(nn.Module):
    def __init__(self, num_dilations, num_streams):
        super(ParallelDilatedConv, self).__init__()
        self.m = num_dilations
        self.streams = [torch.cuda.Stream() for i in range(num_streams)] 
        self.module = nn.ModuleList([nn.Conv2d(1, 1, (3, 3), dilation=2**i, padding=2**i) for i in range(num_dilations)])
                
    def forward(self, input):
        res = []
        for i in range(self.m):
            with torch.cuda.stream(self.streams[i%len(self.streams)]):
                res.append(self.module[i](input))
        return torch.cat(res)


class DilatedConv(nn.Module):
    def __init__(self, num_dilations):
        super(DilatedConv, self).__init__()
        self.m = num_dilations
        self.module = nn.ModuleList([nn.Conv2d(1, 1, (3, 3), dilation=2**i, padding=2**i) for i in range(self.m)])

    def forward(self, input):
        res = []
        for i in range(self.m):
            res.append(self.module[i](input))
        return torch.cat(res)


def time_loop(mod, num_iter, outstr):
    start = time.time()
    for i in range(num_iter):
        mod(im).cpu()
    end = time.time()
    print(outstr.format(end-start))

if __name__ == '__main__':
    import time
    num_iter = 10
    num_conv = 6
    device = 'cuda:0'
    num_streams = 6
    s = torch.cuda.Stream()
    with torch.cuda.stream(s):
        im = torch.rand(1000, 1, 200, 200).to(device, non_blocking=True)


        mod = DilatedConv(num_conv).to(device, non_blocking=True).share_memory()
        time_loop(mod, num_iter, 'Sequential took {}')
        
        mod = ParallelDilatedConv(num_conv, num_streams).to(device, non_blocking=True).share_memory()
        time_loop(mod, num_iter, 'Parallel took {}')
    
    
</code></pre>

pietern: <p>It’s likely that the convs you’re launching are big enough to occupy the entire GPU. When you launch a number of big kernels on different streams, they end up being executed sequentially. Smaller kernels, that use only a small slice of GPU resources, can be parallelized by using multiple streams.</p>
Dimension mismatch between data and nn.Linear tensors?

johnmwu: <p>I’m working through <a href="https://pytorch.org/tutorials/beginner/nlp/deep_learning_tutorial.html" rel="nofollow noopener">this</a> tutorial, but this problem is slightly more general.</p>
<p>In the BoW model there, the input data tensor is of shape (1, 26), but the printed weight matrix (tensor) is of shape (2, 26). I am confused by this, because you cannot multiply a 1x26 matrix with a 2x26 matrix.</p>
<p>Of course, I assume PyTorch does something internally to make everything proper. However, my question is: why does this apparent dimension mismatch exist?</p>

ptrblck: <p>The weight matrix is transposed as before applying the matrix multiplication (<a href="https://github.com/pytorch/pytorch/blob/556af7c19db90465703a9acc0b437cbcfb018ee0/torch/nn/functional.py#L1419" rel="nofollow noopener">line of code</a>), which makes the shapes compatible.</p>

johnmwu: <p>Thanks for the answer. This clears up why it works, but it does not answer the original question.</p>
<p>Why does this dimension mismatch exist? What is the purpose? Is there some invariant that PyTorch wants to maintain (eg. the first axis of any data tensor should index data points). Why not just print the weight matrix as a 26x2 matrix?</p>

ptrblck: <p><a href="https://discuss.pytorch.org/t/why-does-the-linear-module-seems-to-do-unnecessary-transposing/6277/2">This topic</a> might have the answer. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
How to write resnet block for 1 dimensional data

rajeevnarayanan: <p>Newbie alert !<br>
I am experimenting with single dimensional data from a medical device using resnets. Seems like I have to write it from scratch since I have to predict an array of classes.</p>
<p>When I used ConvID for conv, i realized that dimensions dont match exactly with original dimensions ,when I padded it with zeros to match dimensions to do add (), I got bad accuracy .</p>
<p>Any pointers will be helpful.</p>
<p>Thanks</p>
Running dcgan example

mahmoodn: <p>Hi<br>
I have installed pytorch via <code>python3 setup.py install</code>  and the last lines are</p>
<pre><code class="lang-auto">
writing top-level names to torch.egg-info/top_level.txt
writing manifest file 'torch.egg-info/SOURCES.txt'
reading manifest file 'torch.egg-info/SOURCES.txt'
writing manifest file 'torch.egg-info/SOURCES.txt'
Copying torch.egg-info to /usr/local/lib/python3.6/dist-packages/torch-1.2.0a0+08a0ac8-py3.6.egg-info
running install_scripts
Installing convert-caffe2-to-onnx script to /usr/local/bin
Installing convert-onnx-to-caffe2 script to /usr/local/bin
</code></pre>
<p>As I run dcgan example, the cmake command fails</p>
<pre><code class="lang-auto">mahmood@u1804:build$ cmake -DCMAKE_PREFIX_PATH=/usr/local/lib/python3.6/dist-packages/torch/lib/libtorch.so ..
-- The C compiler identification is GNU 7.4.0
-- The CXX compiler identification is GNU 7.4.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
CMake Error at CMakeLists.txt:4 (find_package):
  By not providing "FindTorch.cmake" in CMAKE_MODULE_PATH this project has
  asked CMake to find a package configuration file provided by "Torch", but
  CMake did not find one.

  Could not find a package configuration file provided by "Torch" with any of
  the following names:

    TorchConfig.cmake
    torch-config.cmake

  Add the installation prefix of "Torch" to CMAKE_PREFIX_PATH or set
  "Torch_DIR" to a directory containing one of the above files.  If "Torch"
  provides a separate development package or SDK, be sure it has been
  installed.


-- Configuring incomplete, errors occurred!
See also "/home/mahmood/cactus/pt/pytorch/examples/cpp/dcgan/build/CMakeFiles/CMakeOutput.log".
</code></pre>
<p>How can I fix that?</p>
How to find index of n-th value equal to a specific value

Biao_Zhang: <p>Suppose<br>
<code>a=torch.randint(3, (10,))</code><br>
gives</p>
<pre><code class="lang-auto">tensor([2, 2, 0, 2, 2, 2, 2, 0, 1, 2])
</code></pre>
<p>Now I want to find the index of 3rd value equal to <code>2</code>, which is 3 in this example. Because</p>
<pre><code class="lang-auto">a[0]=2, a[1]=2, a[3]=2
</code></pre>
<p>How can I do this?</p>

sonnguyen: <aside class="quote no-group" data-username="Biao_Zhang" data-post="1" data-topic="48411">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/biao_zhang/48/13267_2.png" class="avatar"> Biao_Zhang:</div>
<blockquote>
<p>a=torch.randint(3, (10,))</p>
</blockquote>
</aside>
<p>I think that one should work<br>
(a==2).nonzero()[2]</p>
Why do I use the same LSTM and get the same output given different inputs

wdp: <p>When I use my own trained LSTM,I take two different input vectors, but you get the same output.<br>
Thank you<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/c/c93b87531aa023c9db5914db3a507dc794b82be6.png" data-download-href="https://discuss.pytorch.org/uploads/default/c93b87531aa023c9db5914db3a507dc794b82be6" title="I%40OO5J_T_%5DG(NL3M%60%40_8X)W"><img src="https://discuss.pytorch.org/uploads/default/original/2X/c/c93b87531aa023c9db5914db3a507dc794b82be6.png" alt="I%40OO5J_T_%5DG(NL3M%60%40_8X)W" data-base62-sha1="sIbvGn3DPjZbjFMnleQ5tWvcVRs" width="690" height="205" data-dominant-color="121212"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">I%40OO5J_T_%5DG(NL3M%60%40_8X)W</span><span class="informations">1688×502 20.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
IndexError: index 68788682753 is out of bounds for dimension 0 with size 6

Yuerno: <p>Hello all. I’ve been trying to train a model with a dice loss function that has two inputs, “logits” and “true” with shapes of [4, 6, 256, 256] and [4, 256, 256] respectively. Unfortunately, the training process crashes with the following error:</p>
<p><code>IndexError: index 68788682753 is out of bounds for dimension 0 with size 6</code></p>
<p>and it crashes on this line of code in particular:</p>
<pre><code class="lang-auto">true_1_hot = torch.eye(num_classes)[true].cuda()
</code></pre>
<p>where <code>num_classes = logits.shape[1]</code></p>
<p>The annoying thing about this error is that it isn’t consistent; in two tests I’ve done, the first time it trains for 5 epochs before crashing with this error, and in the next one it trains for 150 epochs before crashing. Therefore, I can’t figure out how it would go through this exact code multiple iterations of training without any issues, and then decide to crash all of a sudden. The index # it crashes on with that error always remains the same however.</p>
<p>Any help would be greatly appreciated!</p>

albanD: <p>Hi,</p>
<p>It looks like you’ve got some indexing problems. Maybe one sample in your dataset has a wrong value for label? and it appears randomly depending on when this label is drawn by the dataloader?</p>
<p>Also cuda errors are asynchronous so if the error is raised within cuda, you can add <code>CUDA_LAUNCH_BLOCKING=1</code> to make sure the error is raised at the right place.</p>

Yuerno: <p>Hi, thanks for the reply! I turned off the “shuffle” parameter for my dataloader, which I think should now guarantee that each epoch should go through all of the training images sequentially. So far, after a couple epochs of training, no error. What I figured was that if it was a mask image somewhere in the training set causing the issue, having the dataloader go through all of them SHOULD force the error to pop up, right?</p>

albanD: <p>Yes it should.<br>
If you re-enable the shuffling the error happens again? Can you add some printing to check if all the Tensors have the right value.</p>
<p>If you can make a small code sample (~30 lines) that reproduce the problem, that would be very helpful !</p>

Yuerno: <p>I’m honestly not sure, but I may have fixed this issue through <a href="https://discuss.pytorch.org/t/what-can-cause-differences-in-training-a-model-in-windows-vs-ubuntu/48180">fixing another issue I had in another post earlier this week involving file paths for images and masks</a>, and there possibly being mismatches between raw images and their corresponding masks. Perhaps that was causing this indexing issue? At the very least, I haven’t had any crashes since I made those changes to ensure all images and masks are matched correctly on Linux (and Windows), and I haven’t been able to replicate it.</p>
Numerical differences between manually computed gradients and Pytorch autograd

jeandut: <p>I asked this question on stack a while ago:<br>
<aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stackoverflow.com/questions/55299412/numerical-differences-between-manually-computed-gradient-and-pytorch-chain-rule" target="_blank" rel="nofollow noopener">stackoverflow.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/4844184/jean" target="_blank" rel="nofollow noopener">
    <img alt="jean" src="https://www.gravatar.com/avatar/055c6d4adf1d157725f2d920fab121c9?s=128&amp;d=identicon&amp;r=PG&amp;f=1" class="thumbnail onebox-avatar" width="128" height="128">
  </a>
<h4>
  <a href="https://stackoverflow.com/questions/55299412/numerical-differences-between-manually-computed-gradient-and-pytorch-chain-rule" target="_blank" rel="nofollow noopener">Numerical differences between manually computed gradient and PyTorch chain rule</a>
</h4>

<div class="tags">
  <strong>python, deep-learning, pytorch, backpropagation</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/4844184/jean" target="_blank" rel="nofollow noopener">
    jean
  </a>
  on <a href="https://stackoverflow.com/questions/55299412/numerical-differences-between-manually-computed-gradient-and-pytorch-chain-rule" target="_blank" rel="nofollow noopener">12:16PM - 22 Mar 19 UTC</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
Does someone have an explanation ? Are there some operations that are more numerically stable, that I should use ?<br>
What are the order of magnitude in the relative differences that should be expected ? Thanks again.</p>

albanD: <p>Hi,</p>
<p>I haven’t looked very closely to the code, but the first thing is that with floating points, the precision is going to be very very bad. All the tests we do for gradients are actually done in double precision to be sure.</p>

jeandut: <p>Yeah I know that you usually check things like finite differences approximation against autograd in float64 however I thought that since I am computing analytical expression against analytical expression for a very small network it should not matter that much. But I am no expert on single precision calculus and I was wondering if things like the order of the multiply/sum mattered and stuff like that and how easy it was to make things match perfectly.</p>

albanD: <p>float operation are not associative I’m afraid. So it’s going to be impossible to make them match exactly unless you know exactly in which order each op is executed.<br>
From that point, depending on the other values, the small difference might increase quite fast.</p>
<p>Your code looks ok.<br>
I would compute the difference between the prediction and the ground truth once and reuse it.<br>
You can check the intermediary results to see if it’s just an error getting bigger and bigger.</p>

jeandut: <p>OK thanks for your advices !</p>
How to fix the PyTorch installation that can't load the wrapped C libraries?

AvaMAA8: <p>I’m working on google colab ( python 3.6 and GPU), I import torch (1.2.0) with no errors and I use the following to import fastai:</p>
<pre><code class="lang-auto">    import fastai
    print(fastai.__version__)
    from fastai import *
    from fastai.vision import *
</code></pre>
<p>I get the following error:</p>
<p><code>ImportError: /usr/local/lib/python3.6/dist-packages/torchvision/_C.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN3c106Device8validateEv</code></p>
<p>I have tried to install different torch versions like 1.0.0 and work with earlier python versions. I have also tried to install fastai and its dependencies manually using  <code>!pip</code> , but nothing worked.</p>
<p>This is the complete code I used to install torch:</p>
<pre><code class="lang-auto">      from os.path import exists
      from wheel.pep425tags import get_abbr_impl, get_impl_ver, get_abi_tag

    platform = '{}{}-{}'.format(get_abbr_impl(), get_impl_ver(), 
    get_abi_tag())
    cuda_output = !ldconfig -p|grep cudart.so|sed -e 's/.*\.\([0-9]*\)\.\ 
    ([0-9]*\)$/cu\1\2/'
    accelerator = cuda_output[0] if exists('/dev/nvidia0') else 'cpu'

    !pip install torch_nightly -f 
     https://download.pytorch.org/whl/nightly/{accelerator}/torch_nightly.html
    !pip install fastai

     import torch

     print(torch.__version__)
     print(torch.cuda.is_available())
     print(torch.backends.cudnn.enabled)
</code></pre>
<p>And this is the full error message I get:</p>
<pre><code class="lang-auto">    ImportError                                                           
    Traceback (most recent call last)
    &lt;ipython-input-5-4b8b8d8134df&gt; in &lt;module&gt;()
  2 print(fastai.__version__)
  3 from fastai import *
  ----&gt; 4 from fastai.vision import *
    8 frames
  /usr/local/lib/python3.6/dist-packages/torchvision/ops/boxes.py in 
  &lt;module&gt;()
  1 import torch
  ----&gt;  2 from torchvision import _C
  3 
  4 
  5 def nms(boxes, scores, iou_threshold):

  ImportError: /usr/local/lib/python3.6/dist- 
  packages/torchvision/_C.cpython-36m-x86_64-linux-gnu.so: undefined 
  symbol: _ZN3c106Device8validateEv  
</code></pre>
<p>I’m unable to use <code>ImageDataBunch.from_folder</code>  because of the Pytorch installation error and the fastai import error. I get the error  <code>NameError: name 'ImageDataBunch</code>  is not defined when I do.</p>
<p>Note: I did use the same code before and I was able to use fastai and  <code>ImageDataBunch.from_folder</code> with no import errors , but I’m guessing that an update to fastai or torch happened.</p>

HARIKRISHNAN_R: <p>This is happening because the latest version of torchvision i.e., 0.3, has C++ ops that have been compiled with some version of cuda. Pull request <span class="hashtag">#971</span> on torchvision fixes this. I would recommend installing torchvision from source. The instructions can be found on the torchvision github repo.</p>
torch.utils.data.DataLoader, By which order image files are loaded?

m.saber: <p>I am using DataLoader as follows:</p>
<pre><code>train_loader = torch.utils.data.DataLoader(train_folders, batch_size=10, shuffle=False) 
test_loader = torch.utils.data.DataLoader(test_folders, batch_size=10, shuffle=False)
</code></pre>
<p>Imagine that I have 2 folders in the data set as follows:</p>
<p>Class A: A1.png, A2.png. A3.png. …<br>
Class B: B1.png, B2.png. B3.png. …</p>
<p>I would like to know by which order Data Loader loads the image files. As an example, which image files are present in the 1st batch, 2nd batch, etc.</p>
<p>Why do I ask this question?<br>
I am designing CNN for Regression problem not Classification. Every image file has a numeric output. Numerical data are saved in a csv file for all images. I need to relate the numeric data with the images loaded by Data Loader.</p>

Oli: <p>Hi, it depends on the dataset (train_folders) that you are using. I’d advise you to create your custom dataset class. That way you can also return the path of the image and the problem is solved.</p>
<p><a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" rel="nofollow noopener">tutorial</a></p>
DataParallel error when applied to simplest possible architecture: "Module must have its parameters and buffers on device"

hyperdo: <p>I’ve been having a lot of problems with DataParallel. I’ve tried the simplest possible version of DataParallel I can think of, and it still errors out. Any help or advice would be greatly appreciated! This is running on a server with two P100 GPUs.</p>
<pre><code class="lang-auto">In [4]: mlp = nn.DataParallel(nn.Linear(100, 200))
In [5]: mlp(torch.zeros((32, 100)))
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
~/code/projects/mve_sac/core.py in &lt;module&gt;
----&gt; 1 mlp(torch.zeros((32, 100)))
/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    491             result = self._slow_forward(*input, **kwargs)
    492         else:
--&gt; 493             result = self.forward(*input, **kwargs)
    494         for hook in self._forward_hooks.values():
    495             hook_result = hook(self, input, result)
/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py in forward(self, *inputs, **kwargs)
    144                 raise RuntimeError("module must have its parameters and buffers "
    145                                    "on device {} (device_ids[0]) but found one of "
--&gt; 146                                    "them on device: {}".format(self.src_device_obj, t.device))
    147
    148         inputs, kwargs = self.scatter(inputs, kwargs, self.device_ids)
RuntimeError: module must have its parameters and buffers on device cuda:0 (device_ids[0]) but found one of them on device: cpu
</code></pre>

vabh: <p>You need to move the parameters of <code>nn.Linear</code> to <code>cuda</code>. As the error message says, they are currently on <code>cpu</code>.</p>
<pre><code class="lang-auto">linear = nn.Linear(100, 200).cuda()
mlp = nn.DataParallel(linear)
mlp(torch.zeros((32, 100)))</code></pre>
NaN values while going with other form of Sigmoid

Ashish_Bairwa: <p>I’ve tried to implement sigmoid function with it’s altered form. Why i’m facing <strong>NaN</strong> values in the first case?</p>
<p><strong>Image -(Showing torch.exp(x)/1+torch.exp(x))</strong><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/7/735d8e7da2d0d2cd638fe829e5923d9dde22cea3.png" data-download-href="https://discuss.pytorch.org/uploads/default/735d8e7da2d0d2cd638fe829e5923d9dde22cea3" title="rr1"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/7/735d8e7da2d0d2cd638fe829e5923d9dde22cea3_2_690x388.png" alt="rr1" data-base62-sha1="gszlTI2ca5Z0EwbCrY0M0Wppmfx" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/7/735d8e7da2d0d2cd638fe829e5923d9dde22cea3_2_690x388.png, https://discuss.pytorch.org/uploads/default/optimized/2X/7/735d8e7da2d0d2cd638fe829e5923d9dde22cea3_2_1035x582.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/7/735d8e7da2d0d2cd638fe829e5923d9dde22cea3.png 2x" data-dominant-color="F2F3F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">rr1</span><span class="informations">1131×637 76.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

charan_Vjy: <p>Probably an issue with the inputs. Have randomised the inputs and used the sigmoid as you have. Works perfectly fine.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/d/dab4781eca143f5a5480fb8f1a4d66ae5765f502.png" data-download-href="https://discuss.pytorch.org/uploads/default/dab4781eca143f5a5480fb8f1a4d66ae5765f502" title="Py_code"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/d/dab4781eca143f5a5480fb8f1a4d66ae5765f502_2_655x499.png" alt="Py_code" data-base62-sha1="vcKK0xZraChaz1O8YYmu0opXJPY" width="655" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/d/dab4781eca143f5a5480fb8f1a4d66ae5765f502_2_655x499.png, https://discuss.pytorch.org/uploads/default/optimized/2X/d/dab4781eca143f5a5480fb8f1a4d66ae5765f502_2_982x748.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/d/dab4781eca143f5a5480fb8f1a4d66ae5765f502.png 2x" data-dominant-color="EEEFEF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Py_code</span><span class="informations">1141×870 79.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
How to use mkldnn on conda installation?

HITerStudy: <p>If I use conda tool, that is, “conda install mkldnn” to install mkldnn, then I don`t know what to do in the next step for the link between mkldnn and pytorch, please give me some instructions if possible?Thanks a lot!</p>

LeviViana: <p>I guess you should first inspect if a pre-compiled pytorch is already using mkldnn:</p>
<pre><code class="lang-python">import torch
print(*torch.__config__.show().split("\n"),sep="\n")
</code></pre>
<p>If <code>USE_MKLDNN=ON</code>, then you are already using it. Otherwise, the other simplest way I have in mind if you want to bind Pytorch and mkldnn is to build it from source.</p>

HITerStudy: <p>Thanks a lot, I will try it to make sure that the mkldnn is working!</p>
How to set value of a Tensor with numpy array

david-leon: <p>I’m looking for <code>tensor.set_value()</code> and <code>tensor.get_value()</code> equivalence in pytorch. For <code>tensor.get_value()</code> it seems <code>Tensor.numpy()</code> meets the demand, but for <code>tensor.set_value()</code> I got no clue.</p>
<p>So, is there any way to set value of a pytorch tensor with numpy array? <code>torch.from_numpy()</code> returns a new Tensor, so it’s not the right answer.</p>

david-leon: <p>I just realized that what <code>Tensor.numpy()</code> returns is a pointer, with which <code>Tensor</code>'s value can be modified. It works for cpu tensors, not tested yet for gpu tensors.</p>

LeviViana: <p>Is this what you are looking for ?</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/f/ff2d92543fe47bde87a8fef7ea62c1f0e491c5dd.png" alt="image" data-base62-sha1="AppohyguHpyQt0IjDMZyqVUvGON" width="277" height="97"></p>
Using Pytorch for an interactive demo

Nadav_Bhonker: <p>Is there any reference how I’d get by creating an interactive demo in the browser using a Pytorch model?<br>
Maybe something like <a href="https://nvlabs.github.io/SPADE/demo.html" rel="nofollow noopener">this</a> or <a href="https://magenta.tensorflow.org/demos/performance_rnn/index.html#2%7C2,0,1,0,1,1,0,1,0,1,0,1%7C1,1,1,1,1,1,1,1,1,1,1,1%7C1,1,1,1,1,1,1,1,1,1,1,1%7Cfalse" rel="nofollow noopener">this</a></p>

alex.veuthey: <p>You have to see the PyTorch model as a building block in the whole demo. There’s a backend and a frontend and many things happening in those demos, and this forum is not really dedicated to building websites <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=9" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"></p>
<p>But of course you can create a python backend (bottle, flask…) and any frontend you like, or use full-stack frameworks, and use PyTorch code inside when processing client requests. Or even another backend that can call python code <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Efficient computation of median ignoring zeros (only torch operations)

DrJayLight: <p>Hi!</p>
<p>GIven a 2D tensor, i’d like to compute the median for each row in that tensor in the most efficient way possible, but the caveat here is that I’d like to do this ignoring any zeros for the computation of the median.</p>
<p>So for example if my tensor is,</p>
<pre><code class="lang-auto">[ [3, 1, 2], 
  [0, 0, 1],
  [0, 0, 0] ]
</code></pre>
<p>After computing the median I would expect to get the tensor with output <code>[2, 1, 0]</code>.</p>
<p>My current approach is as follows:</p>
<pre><code class="lang-auto">for (int i = 0; i &lt; tensor.size(0); ++i) {  // go through each row in tensor
    noZeroInds = torch::nonzero(tensor[i].view(-1)).view(-1);
    outputTensor[i] = torch::median(tensor[i].index_select(0, noZeroInds))
</code></pre>
<p>So as you can see, I am going through each row in the tensor, selecting the nonZero indices, computing the median for the selected row, and finally storing this value to some output tensor.</p>
<p>So my question is, is there any way to do this in a more efficient vectorized fashion using torch operations?</p>
Version 1.1.0 with high %CPU

Leigang_Qu: <p>Hi~ I have found that my process occupies higher CPU(about 400% or higher) with version 1.1.0 than with version 1.0.1 or version 0.4.1(about 100%).<br>
The above phenomenon happens only on setting pip_memory = True in DataLoader. When setting pip_memory = Flase, the CPU is about 100%. Besides, num_workers is 6.<br>
Could you please tell me how to configue my pytorch 1.1.0 to maintain CPU not too high with pip_memory still True ?</p>
ImportError: No module named fb

Ryan95: <p>Hi, when i run caffe2/quantization/server/utils.py, i get this error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "caffe2/quantization/server/conv_dnnlowp_op_test.py", line 8, in &lt;module&gt;
    from caffe2.quantization.server import utils as dnnlowp_utils
  File "/home/ryankang/workspace/projects/pytorch/build/caffe2/quantization/server/utils.py", line 8, in &lt;module&gt;
    from caffe2.python.fb import hardcode_scale_zp
ImportError: No module named fb
</code></pre>
<p>How to deal it? thanks!</p>
Efficiently computing max drawdown

evan: <p>I’m working with financial data representing a portfolio, and one of the things I want to optimize for is <a href="http://www.investopedia.com/terms/d/drawdown.asp" rel="nofollow noopener">max drawdown</a>. The drawdown at any point is the difference of the max portfolio value so far minus the current value.</p>
<p>Unoptimized Python code to compute this might look like this:</p>
<pre><code class="lang-python">def max_drawdown(vec):
    drawdown = 0.
    max_seen = vec[0]
    for val in vec[1:]:
        max_seen = max(max_seen, val)
        drawdown = max(drawdown, 1 - val / max_seen)
    return drawdown
</code></pre>
<p>This can be improved using <code>numpy.maximum.accumulate</code>:</p>
<pre><code class="lang-auto">def max_drawdown(vec):
    maximums = np.maximum.accumulate(vec)
    drawdowns = 1 - vec / maximums
    return np.max(drawdowns)
</code></pre>
<p>Translating the naive for-loop implementation to Pytorch is straightforward, but it causes my model to become incredibly slow during autograd. I want to optimize the code, ideally with something that looks similar to the <code>np.maximum.accumulate</code> version I gave above.</p>
<p>It is possible to do this with Pytorch? If so, how?</p>

SS_SS: <p>HI,<br>
What does “vec” stand for?</p>
Can you specify convolution or corelation?

blackbirdbarber: <p>For <a href="https://pytorch.org/docs/stable/nn.html#conv2d" rel="noopener nofollow ugc">conv2D</a>  can you specify <a href="https://stackoverflow.com/a/35417934/5884955" rel="noopener nofollow ugc">convolution or correlation</a> .</p>
<p>From <a href="https://www.deeplearningbook.org/contents/convnets.html" rel="noopener nofollow ugc">source</a>:</p>
<blockquote>
<p>Many machine learning libraries implement cross-correlation but call it convolution.</p>
</blockquote>
<p>Is this the case with PyTorch? Does it matter. I haven’t found corelate2D class.<br>
Comments…</p>

ptrblck: <p>PyTorch uses cross-correlation.<br>
It shouldn’t make any difference, since the kernels will just be flipped for a convolution.</p>
<p>However, if you are trying to apply some manually defined kernels for image processing (e.g. edge detectors), you should flip them to get the desired results.<br>
That being said, it should not make a difference in your training, if you just use randomly initialized kernels.</p>