akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
The application will happen exception(0xc0000005) when it unloads dll using libtorch 1.2

username: weiqiangguo
cooked: <p>OS: Win10 1809<br>
LibTorch 1.2<br>
CUDA 10</p>
<p>The application will happen exception(0xc0000005) when it unloads dll using libtorch 1.2.<br>
The windows Logs as the follow:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/b/7bd84ec6055880a33ddc669c0cd966fe08566b8b.png" data-download-href="https://discuss.pytorch.org/uploads/default/7bd84ec6055880a33ddc669c0cd966fe08566b8b" title="winlog_1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/b/7bd84ec6055880a33ddc669c0cd966fe08566b8b_2_677x500.png" alt="winlog_1" data-base62-sha1="hFAai598MSp1Sfh7YU72e2JUcRR" width="677" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/b/7bd84ec6055880a33ddc669c0cd966fe08566b8b_2_677x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/b/7bd84ec6055880a33ddc669c0cd966fe08566b8b_2_1015x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/b/7bd84ec6055880a33ddc669c0cd966fe08566b8b.png 2x" data-dominant-color="E6E6E8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">winlog_1</span><span class="informations">1171×864 73.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: peterjc123
cooked: <p>Did you copy all the DLLs in <code>LibTorch\lib</code> to where the executable locates?</p>

username: weiqiangguo
cooked: <p>Yes, these all dlls have been copied to the work directory.<br>
Have you ever had the same problem?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/6/16e0203077749633af54701a71799e11e9bcc95e.png" data-download-href="https://discuss.pytorch.org/uploads/default/16e0203077749633af54701a71799e11e9bcc95e" title="dpcode_dll"><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/6/16e0203077749633af54701a71799e11e9bcc95e.png" alt="dpcode_dll" data-base62-sha1="3gmGSfuR4yLD4gxnChneYFjR6R8" width="608" height="500" data-dominant-color="F7F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">dpcode_dll</span><span class="informations">921×757 51.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Can i install pytorch 1.1.0 from source, with NVIDIA CUDA 8.0.44?

username: WMF1997
cooked: <p>hello:<br>
I want to install pytorch 1.1.0, while my Linux’s CUDA’s version is 8.0.44 (I checked it from nvcc --version)<br>
I find out that pytorch 1.1.0 does not have the .whl files that meets my need. I have to install it from source (with GPU support).</p>
<p>So I want to ask if I can install pytorch 1.1.0 from source, with NVIDIA CUDA 8.0’s GPU acceleration support. I know that CUDA 8.0 is very old now.</p>
<p>yours sincerely,<br>
MingFei Wang.</p>

username: ptrblck
cooked: <p>As far as I know, the min CUDA version is CUDA9 to install PyTorch from source.<br>
Do you really need this older version or could you update to a newer one?</p>

username: WMF1997
cooked: <p>Thank you very much~ And I decide to update CUDA.</p>
Best way to debug DistributedDataParallel code?

username: Mxbonn
cooked: <p>Hey,</p>
<p>I’m having an issue that my code randomly hangs when using DistributedDataParallel.<br>
It is completely random when this occurs, and it does not always occur.<br>
I suspect that it has something to do with the DistributedDataParallel as out of the 4 gpu’s I’m using, 3 are reporting to be using 100% of that gpu and 1 is completely idle.</p>
<p>What is the best way for me to debug what is going on as I’m getting no errors?</p>

username: mrshenli
cooked: <p>Looks like the program somehow desynchronized (different processes wants to sync different amount of parameters or run different numbers of iterations). Unfortunately, DDP does not have a debug mode for now. Can you share some code snippet? Does your code tries to handle any errors in backward pass on its own (say catch OOM and rertry)? Does all processes see exactly the same amount of input data?</p>

username: Yaroslav_Bulatov
cooked: <p>I’ve had to deal with similar issues, you should feel lucky you only have 4 processes and not 64 <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>The fact that 3 are in 100% utilization means that are inside nccl sync operation like the one at the end of backward .backward(), while the 4th one is doing something non-GPU related, like waiting for user input.</p>
<p>The general strategy is to look at stack trace of the “odd one out” process. You can get C++ stack trace by doing "gdb -p " and “bt” or “thread apply all bt”</p>
<p>With a bit more work, you can get Python stack. This requires modifying client code. For instance, I run <a href="https://gist.github.com/yaroslavvb/5de8ad332f2e37aa542e69e208476433" rel="nofollow noopener">install_pdb_hander</a> on init of my processes. It allows me to break into PDB on CTRL+\ and look what the current process is doing.</p>
<p>When using distributed launcher, this will only send CTRL+\ to the launcher process, so for this to get you stack trace of arbitrary worker you could need to modify your launching procedure to run in 4 worker in 4 different tmux windows</p>

username: Yaroslav_Bulatov
cooked: <p>related issue <a href="https://github.com/pytorch/pytorch/issues/27757" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/27757</a></p>
Synchronizing/pausing all processes but one with DistributedDataParallel

username: Art
cooked: <p>Hello,<br>
I’m working with the code example from the <a href="https://github.com/pytorch/examples/tree/master/imagenet#multi-processing-distributed-data-parallel-training" rel="nofollow noopener">official ImagNet distributed tutorial</a>.<br>
Basically, the code uses <code>torch.multiprocessing.spawn(main_worker)</code> to run a copy of the “main worker” function on each GPU.<br>
Then, in each worker the <code>dist.init_process_group</code> command is run and both the model and dataset/dataloader are created and cast into <code>torch.nn.parallel.DistributedDataParallel(model)</code> and <code>torch.utils.data.distributed.DistributedSampler(dataset)</code>.</p>
<p>My problem is that after every epoch I want to modify and rewrite all the data in the dataset, I thought the most straightforward way was to run this shuffling in only one of the nodes inside of an “if” like <code>if args.multiprocessing_distributed and args.rank == 1:</code> so not all the nodes would perform the shuffling simultaneously, similarly to <a href="https://github.com/pytorch/examples/blob/master/imagenet/main.py#L252" rel="nofollow noopener">the 252 line of the code</a>.</p>
<p>The operation of rewriting the dataset takes long time (&gt;10 minutes). This created a problem where processes which didn’t perform the shuffling were trying to access data while it’s being rewritten by the node that modifies the data.</p>
<p>Is there any suggested way of making the rest of the processes wait for the one process to finish the modification and writing of the data before continuing the training?</p>
<p>I found <a href="https://pytorch.org/docs/stable/distributed.html#torch.distributed.barrier" rel="nofollow noopener">this “barrier” method</a> from the pytorch distributed package.<br>
And also this <a href="https://docs.python.org/3/library/multiprocessing.html#synchronization-primitives" rel="nofollow noopener">section about synchronization</a> in python’s documentation.<br>
But because the error I get isn’t 100% reproducible (in some runs it appears 10 minutes after start in some runs it appears hours into the run) I can’t really test the those implementations and I couldn’t find any easy to follow examples.</p>
<p>Any suggestions would be appreciated <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=9" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"></p>

username: mrshenli
cooked: <p><code>dist.barrier()</code> should help to block all processes in the group until everyone has reached the same barrier. What error did you see after adding a barrier?</p>
With torch.no_grad(): - can I include optimizer, backward and etc?

username: andreiliphd
cooked: <p>Can I do with torch.no_grad():</p>
<pre><code class="lang-auto">        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
</code></pre>

username: ptrblck
cooked: <p>No that won’t work, as this disables gradient calculation.<br>
From the <a href="https://pytorch.org/docs/stable/autograd.html?highlight=no_grad#torch.autograd.no_grad">docs</a>:</p>
<blockquote>
<p>Context-manager that disabled gradient calculation.<br>
Disabling gradient calculation is useful for inference, when you are sure that you will not call <code>Tensor.backward()</code> . It will reduce memory consumption for computations that would otherwise have requires_grad=True. In this mode, the result of every computation will have requires_grad=False, even when the inputs have requires_grad=True.</p>
</blockquote>
<p>What is your intention?</p>

username: andreiliphd
cooked: <p>I was just wanted to implement dataloader on the fly but I found a better solution without much headache. It’s a DataLoader from PyTorch that handles all the problems of creating target and input by providing a folder name with folders as categories.</p>

username: tuf22191
cooked: <p>Yeah I know the one you are talking about - DL4J also has a similar dataloading capability if you ever need it.</p>
How to fuse feature maps to a big feature map for Conv layers

username: Mylinda
cooked: <p>Hi guys!</p>
<p>I meet some problems and hope I could get help from you. My problems are as below:</p>
<p>I wanna design a DCNN to integrate feature maps from different DCNNs for classification. My problem is how to fuse these feature maps? My initial idea is to concatenate these feature maps and then put the big feature map to convolutional layers. However, the number of feature maps from different DCNNs is variable. This makes the in_channels of the Con2d is uncertain after concatenating them.</p>
<p>According to the solution to cope with variable image size, I wanna try the AdaptiveAvgPooling. But it seems this solution need to set the parameter of in_channels of Con2d to None, which is impossible. Right?</p>
<p>Is there any other good solutions? Thank you in advance!</p>

username: ptrblck
cooked: <p>Your idea of concatenating the feature maps and then pass as a new volume to a conv layer sounds plausible.<br>
Could you explain your concerns a bit and why you have a variable number of feature maps?<br>
Do you mean the number of channels by that statement or the spatial size?<br>
In the latter case, you could use e.g. (adaptive) pooling layers to create activation maps of the same spatial size before concatenating them.</p>
Training Loss: nan

username: Arvind_Subramaniam
cooked: <p>I am training a simple conv layer on cifar10 and I keep getting a high loss during training:</p>
<pre><code class="lang-auto">Epoch 1/50 - Training loss: nan
Epoch 2/50 - Training loss: nan
</code></pre>
<p>Conv Network</p>
<pre><code class="lang-auto">import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):
  def __init__(self):
    super(Net, self).__init__()
    self.conv1 = nn.Conv2d(3, 6, 5)
    self.pool = nn.MaxPool2d(2, 2)
    self.conv2 = nn.Conv2d(6, 16, 5)
    self.conv3 = nn.Conv2d(16, 32, 5)
    self.conv4 = nn.Conv2d(32, 64, 5)
    self.conv5 = nn.Conv2d(64, 64, 1)
    self.fc1 = nn.Linear(64 * 1 * 1, 120)
    self.fc2 = nn.Linear(120, 84)
    self.fc3 = nn.Linear(84, 10)

  def forward(self, x):
    x = self.pool(F.relu(self.conv1(x)))
    x = F.relu(self.conv2(x))
    #print(x.shape)
    x = F.relu(self.conv3(x))
    x = F.relu(self.conv4(x))
    x = self.pool(F.relu(self.conv5(x)))
    x = x.view(-1, 64 * 1 * 1)
    x = F.relu(self.fc1(x))
    x = F.relu(self.fc2(x))
    x = self.fc3(x)
    return x
</code></pre>
<p>Here are the training and validation functions:</p>
<pre><code class="lang-auto">def train_conv(model):
  minimum = 1000
  model = model.cuda()
  optimizer = optim.SGD(model.parameters(), lr=0.003, momentum=0.9)
  criterion =nn.NLLLoss()
  epochs = 50
  for epoch in range(epochs):  # loop over the dataset multiple times
    running_loss = 0.0
    for inputs, labels in train_loader:
      # get the inputs; data is a list of [inputs, labels]
      inputs, labels = inputs.cuda(),labels.cuda()
      # zero the parameter gradients
      optimizer.zero_grad()

      # forward + backward + optimize
      outputs = model(inputs)
      loss = criterion(outputs, labels)
      loss.backward()
      optimizer.step()

          # print statistics
      running_loss += loss.item()
    print('[%d, %5d] loss: %.3f' %
                    (epoch + 1, i + 1, running_loss / len(train_loader)))
  return (running_loss/len(train_loader))
</code></pre>
<pre><code class="lang-auto">def validation_conv(model):
  correct = 0
  total = 0
  model = model.cuda()
  with torch.no_grad():
    for data in val_loader:
      images, labels = data
      images, labels = images.cuda(),labels.cuda()
      outputs = model(images)
      _, predicted = torch.max(outputs.data, 1)
      total += labels.size(0)
      correct += (predicted == labels).sum().item()

  print('Accuracy of the network on the 10000 test images: %d %%' % (
      100 * correct / total))
</code></pre>
<p>I am not sure how to change the train_conv function to get the correct training loss.</p>

username: ptrblck
cooked: <p><code>nn.NLLLoss</code> expects log probabilities as the model outputs, so use <code>F.log_softmax</code> on your output before passing it to the criterion or alternatively keep your current model architecture and use <code>nn.CrossEntropyLoss</code> instead, which will internally use <code>F.log_softmax</code> + <code>nn.NLLLoss</code>.</p>
Dataloader for prepossessed images pytorch

username: Frida
cooked: <p>I am doing some pre-processing algorithm on cifar-10 images which takes a lot of time. In order not to go through this process each time, It is preferred to save these images efficiently. It is about 10000 images, same as cifar-10.<br>
I couldn’t find a coherent answer for that, how would you recommend to save these images in a separate dataloader?</p>

username: paganpasta
cooked: <p>You can save the pre-processed images onto the disk and load it using a separate Dataset instance</p>
Computing derivatives of matrices for Gaussian Processes

username: jmaronas
cooked: <p>Hi.</p>
<p>Can I compute matrix derivatives with PyTorch. I am programming some Gaussian processes, setting the hyperparameters by marginal likelihood and I was wondering if the derivatives can be taken automatically with PyTorch.</p>
<p>For instance a simple example would be computing the derivative of:</p>
<pre><code class="lang-auto">N=100
val=teta^2*numpy.eye(N)

</code></pre>
<p>The derivative is:</p>
<pre><code class="lang-auto">N=100
val=2*teta*numpy.eye(N)
</code></pre>
<p>Thanks</p>
Libtorch C++ static linking requires aten::mul operator registration

username: marchss
cooked: <p><a href="https://github.com/pytorch/pytorch/issues/14367" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/14367</a> also mentioned this problem.</p>
<p>Due to some limitation,  <code>-Wl,--whole-archive</code> option cannot be used when linking. The executable binary generated fails to run, with  error “DeviceGuardImpl for cpu is not available”.</p>
<p>Below are two lines added to register the Device. There are new errors with <code>aten::mul</code> missing. After checking the source code under jit/ and aten/, I did not get a clue of how to register all operators.</p>
<p>Could someone please provide some standard initialization code for static library users?</p>
<pre><code class="lang-auto">#include &lt;torch/script.h&gt; // One-stop header.
**#include &lt;ATen/detail/CPUGuardImpl.h&gt;  //ADDED first line**


#include &lt;iostream&gt;
#include &lt;memory&gt;

int main(int argc, const char* argv[]) {
  if (argc != 2) {
    std::cerr &lt;&lt; "usage: example-app &lt;path-to-exported-script-module&gt;\n";
    return -1;
  }

  **C10_REGISTER_GUARD_IMPL(CPU, at::detail::CPUGuardImpl); //ADDED second line**

  // Deserialize the ScriptModule from a file using torch::jit::load().
  std::shared_ptr&lt;torch::jit::script::Module&gt; module = torch::jit::load(argv[1]);

  assert(module != nullptr);
  std::cout &lt;&lt; "ok\n";
}

</code></pre>
<p>Below is the error message:</p>
<pre><code class="lang-auto">terminate called after throwing an instance of 'torch::jit::script::ErrorReport'
  what():  
unknown builtin op: aten::mul
Could not find any similar ops to aten::mul. This op may not exist or may not be currently supported in TorchScript
:

def mul(a : float, b : Tensor) -&gt; Tensor:
  return b * a
         ~~~~~ &lt;--- HERE
def add(a : float, b : Tensor) -&gt; Tensor:
  return b + a
def ne(a : float, b : Tensor) -&gt; Tensor:
  return b != a
def eq(a : float, b : Tensor) -&gt; Tensor:
  return b == a
def lt(a : float, b : Tensor) -&gt; Tensor:
  return b &gt; a
def le(a : float, b : Tensor) -&gt; Tensor:
Aborted
</code></pre>

username: marchss
cooked: <p>Anybody knows? Please help.</p>

username: divinho
cooked: <p>I have the same issue: <code>Unknown builtin op: aten::mul</code></p>
<p>Created an issue <a href="https://github.com/pytorch/pytorch/issues/27726" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/27726</a></p>
How to do batch filling?

username: Steve_Tod
cooked: <p>Hi,</p>
<p>I need to fill images with other image patches during training. Since I’m training with mini-batch, is there any efficient way to do this?</p>
<p>For example, I have a mini-batch of images of size [B, 3, 128, 128]. I also have patches of size [B, 4, 3, 32, 32], where 4 is the number of patches. Besides, I have the bounding box indicating the location of patches of size [B, 4, 4]. How can I fill in the patches in a batch-wise way? You can ignore the overlap between patches.</p>
<p>Thanks!</p>

username: ptrblck
cooked: <p>Could you explain a bit more about how you’ve defined the bounding boxes?<br>
Does each patch have 4 values, corresponding to the upper left x, y and lower right x, y coordinates?</p>

username: Zhihao_Wu
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, you’re right. I answer this question on stackoverflow, <a href="https://stackoverflow.com/questions/58245762/how-to-do-batch-filling-in-pytorch/58246234#58246234" rel="nofollow noopener">How to do batch filling in pytorch</a>. But looks like for-loop isn’t very efficient. Any ideas of improvement?</p>

username: ptrblck
cooked: <p>I think the problematic part trying to get rid of the loops would be the batched way of creating the range indexing for batches.<br>
Probably creating the linear indices beforehand and index the result tensor directly would work.<br>
However, how performance critical is this code?<br>
E.g. if you just run it once in your script, I wouldn’t invest a lot of time improving its performance, but focus on other parts of the code.</p>

username: Steve_Tod
cooked: <p>Thanks guys! Actually I just use <code>for</code> loop suggested by <a class="mention" href="/u/zhihao_wu">@Zhihao_Wu</a> . Because I find out that my GPU can only afford batch size of 8. So <code>for</code> loop do not decrease the performance.</p>
Apply criterion on Multi-class Multi-target and separated classifiers+

username: lmbk
cooked: <p>Hi !</p>
<p>I’m new to NLP and transformers. I have readen documentation on this (BERT, Attention is All you need, …).</p>
<p>I’m trying to implement a Bert model that I want to fine tune which compute two different classifications :</p>
<ul>
<li>first one : 2 classes</li>
<li>second classifier : 5 classes</li>
</ul>
<p>I have a my labels as :</p>
<pre><code class="lang-auto">\y : 

tensor([[[[1],
          [0]]],


        [[[1],
          [0]]]])
</code></pre>
<p>The outputs after model(x) :</p>
<pre><code class="lang-auto">Output : 

[tensor([[ 0.1207, -0.2359],
        [ 0.4030, -0.0475]], grad_fn=&lt;AddmmBackward&gt;), tensor([[ 0.1071,  0.1679,  0.0090, -0.7056, -0.1793],
        [ 0.1295, -0.0781,  0.3041, -0.6385,  0.1090]],
       grad_fn=&lt;AddmmBackward&gt;)]

</code></pre>
<p>As you can see, I have 2 classifier of size 2 and 5. and the labels corresponds to the index of each classifier output.</p>
<p>If I apply criterion as :</p>
<pre><code class="lang-auto">for i in range(2) : #for each label
     loss+= criterion(output[i],y[i])
</code></pre>
<p>For i = 0, I print these two output[0] and y[0] :</p>
<pre><code class="lang-auto">output[i] :

tensor([[ 0.4083, -0.1396],
        [ 0.4059, -0.3079]], grad_fn=&lt;AddmmBackward&gt;)
y[i] :

tensor([[[1],
         [1]]])
</code></pre>
<p>I got the error :<br>
<code>ValueError: Expected input batch_size (2) to match target batch_size (1).</code></p>
<p>I’m stuck on this and I have the impression of forbid something about y format but I’m not sure. Someone could help me on this one ? Thanks you</p>
Create a new Sparse tensor out of several small sparse tensor

username: zcajiayin
cooked: <p>Hi,</p>
<p>I have two sparse (or even more but 2 for demonstration) float tensors, Say,</p>
<pre><code class="lang-auto">i_A = torch.LongTensor([[0, 1], [0, 0]])
v_A = torch.FloatTensor([3, 4])
A = torch.sparse.FloatTensor(i_A, v_A, torch.Size([2,2])).to_dense()
A = [[3, 4],
     [0, 0]]

i_B = torch.LongTensor([[0, 1], [0, 1]])
v_B = torch.FloatTensor([6, 6])
B = torch.sparse.FloatTensor(i_B, v_B, torch.Size([2,2])).to_dense()
B = [[6, 0],
     [0, 6]]
</code></pre>
<p>Now I would like to create a new sparse tensor <code>C</code> in which <code>A</code> and <code>B</code> lie on the diagonal, with everywhere else filled with <code>0</code>:</p>
<pre><code class="lang-auto">C = [[3, 4, 0, 0],
     [0, 0, 0, 0],
     [0, 0, 6, 0],
     [0, 0, 0, 6]]
</code></pre>
Is it possible to access the model replicas in `torch.nn.DataParallel()` between `forward()` and `.backward()` call?

username: ThibaultGROUEIX
cooked: <p>Hi,<br>
I am trying to design backward hooks that depend on the loss.<br>
In the single GPU version, i modify the hook between the forward and backward pass depending on the loss value.<br>
I would like to scale to multi-gpu and modify each replicated hook depending on the loss of each GPU.<br>
To do so, Is it possible to access the model replicas in <code>torch.nn.DataParallel()</code> between <code>forward()</code> and  <code>.backward()</code> call?<br>
Thanks for helping <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=9" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:"></p>

username: mrshenli
cooked: <p>nn.DataParallel does not expose replicas, but you could make some changes to it locally or copy the data_parallel.py code to make replicas a member field (<a href="https://github.com/mrshenli/pytorch/blob/0471b1c270cac2aa301f47994d4141d9c7612fc6/torch/nn/parallel/data_parallel.py#L151" rel="nofollow noopener">this line</a>)</p>
<p>It might be easier to install hooks with DistributedDataParallel (DDP) though, as DDP is not making new copies of models in every iteration. So that whatever hooks installed to the original module should still be valid with DDP.</p>

username: ThibaultGROUEIX
cooked: <p>I’ll take a look at DistributedDataParallel first, it looks the cleanest thing to do.<br>
Thanks for your help <a class="mention" href="/u/mrshenli">@mrshenli</a> !</p>
LSTM not learning in pytorch but working in keras

username: vipin14119
cooked: <p>Hi,<br>
I am facing a weird problem, let me throw in the context first.<br>
I am developing a time series forecasting model using LSTM, basically many to one stacked LSTM. so i used keras first for fast testing, below is my keras model</p>
<pre><code class="lang-auto">model = Sequential()
    model.add(LSTM(100, activation="relu", input_shape=input_shape, return_sequences=True))
    model.add(LSTM(50, activation="relu"))
    model.add(Dropout(0.3))
    model.add(Dense(out_shape))
    model.compile(optimizer=Adam(lr=0.005), loss='mse')
</code></pre>
<p>When i trained this model with above setting i got loss of 0.0035 on my normalized data.</p>
<p>So for more flexiblity i tried using pytorch and made this below model</p>
<pre><code class="lang-auto">class Generator(nn.Module):

    def __init__(self, input_dim, output_dim=1, batch_size=1):
        super(Generator, self).__init__()
        self.input_dim = input_dim
        self.output_dim = output_dim
        self.batch_size = batch_size
        self.hidden_size = 50

        self.lstm_l1 = nn.LSTM(100, input_size=self.input_dim, bidirectional=False, num_layers=1)
        self.lstm_l2 = nn.LSTM(hidden_size=50, input_size=100, bidirectional=False)
        self.fc1 = nn.Linear(50, self.output_dim)

    def forward(self, x):

        x, (h1, c1) = self.lstm_l1(x)
        x, (h2, c2) = self.lstm_l2(x, (h1, c1))
        x = self.fc1(x[-1])  # because i want one output only per input, so picking latest time output
        return x
</code></pre>
<p>I used same settings for training for above model learning rate and loss function and it also gave me around same loss value 0.0031 BUT when i compared the output and plotted a curve , the learned curve was almost straight line but whereas the keras model was able to learning properly.<br>
Can someone please tell me what am i doing wrong.<br>
I reshaped the data into proper shape as required by pytorch lstm</p>
<p><code>Input shape for keras: (292, 20, 1) # batch size, sequence length, num features</code><br>
<code>Input shape for pytorch: (20, 292, 1) . # as required by default lstm</code><br>
<code>Output shape by pytorch lstm: (292, 1) . #means for every 20 step i want 1 step further prediction</code></p>
<p>PS: i tried changing learning rate , hidden units, layers, BatchNorm for lstm but it remains around the same for pytorch.</p>
<p>Please someone guide me.</p>

username: vipin14119
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Can you please give some insight on this ?</p>
How to initialize mean and variance of BatchNorm2d?

username: han.liu
cooked: <p>I’m transforming a TensorFlow model to Pytorch. And I’d like to initialize the mean and variance of BatchNorm2d using TensorFlow model.<br>
I’m doing it in this way:</p>
<pre><code class="lang-auto">bn.running_mean = torch.nn.Parameter(torch.Tensor(TF_param))
</code></pre>
<p>And I get this error:</p>
<pre><code class="lang-auto">RuntimeError: the derivative for 'running_mean' is not implemented
</code></pre>
<p>But is works for <code>bn.weight</code> and <code>bn.bias</code>. Is there any way to initialize the mean and variance using my pre-trained Tensorflow model? Is there anything like  <code>moving_mean_initializer</code>  and  <code>moving_variance_initializer</code>  in Pytorch?<br>
Thanks!</p>

username: ptrblck
cooked: <p>Could you try to assign a <code>torch.tensor</code> instead of an <code>nn.Parameter</code>, since the running estimates do not require gradients?</p>

username: han.liu
cooked: <p>WoW! It worked! Thank you! By the way, if I freeze bn parameters (stop bn from updating), the <code>running_mean</code> and <code>running_var</code> will not change. The <code>running_mean</code> and <code>running_var</code> will be saved in model directly. Am I right? Sorry for bad English expression.</p>

username: han.liu
cooked: <p>WoW! It worked! Thank you! By the way, if I freeze bn parameters (stop bn from updating), the  <code>running_mean</code>  and  <code>running_var</code>  will not change. The  <code>running_mean</code>  and  <code>running_var</code>  will be saved in model directly. Am I right? Sorry for bad English expression.</p>

username: ptrblck
cooked: <p>It depends on what you mean by “freezing”.<br>
To use the running estimates without updating, you could simply call <code>.eval()</code> on the batchnorm layer.<br>
If you would like to freeze the affine parameters (<code>weight</code> and <code>bias</code>), you would need to set their <code>requires_grad</code> attribute to <code>False</code>.</p>

username: han.liu
cooked: <p>Thanks. I meant <code>requires_grad</code>.</p>
Is there a way to share network parameters and gradients across multiple processes?

username: halahup
cooked: <p>I am trying to run the network training in multiple processes. However, the gradients that are computed after the loss is back-propagated are available only in that process not outside, even when the process joins.</p>
<p>Here is a minimal reproducing example:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.multiprocessing as mp

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        
        self.net = nn.Sequential(
            nn.Linear(3, 4),
            nn.ReLU(),
            nn.Linear(4, 1)
        )
    
    def forward(self, x):
        return self.net(x)

def compute_something(network: nn.Module, pid: int):
    print(f'Running PID: {pid}')
    inputs = torch.randn(size=(5, 3))
    logits = network(inputs)
    loss = -1 * torch.mean(torch.pow(logits, 2))
    loss.backward()
    print('INSIDE' + '#' * 10)
    for param in network.parameters():
        print(f'Param: {param}')
        print(f'Grad: {param.grad}')
    print('#' * 10)
    
def main():
    net = Net()
    net = net.share_memory()
    
    processes = [mp.Process(target=compute_something, args=(net, pid, )) 
             for pid in range(2)]
    
    for p in processes:
        p.start()

    for p in processes:
        p.join()
        
    for param in net.parameters():
        print(f'Param: {param}')
        print(f'Grad: {param.grad}')
        
if __name__ == '__main__':
    main()

</code></pre>
<p>In the internal case the gradients are computed and assigned to tensors, but in the external case, the gradients are <code>None</code>. I appreciate any help.</p>

username: halahup
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Do you have any pointers on this? Any help is appreciated.</p>
Padding for conv2d with kernel size 2x2

username: XW324
cooked: <p>Well apparently, to get an output with the same height and width as the input, padding is always needed when the conv2d kernel size is 2. But the pading is only needed in the begining or at the end, not both. I could pad on both sides and then remove the begining or the end afterwards. But is it gonna impact the runtime? Is there an efficient way to do padding on one side only?</p>

username: ptrblck
cooked: <p><code>F.pad</code> is able to pad on each dimension separately and might be useful in your use case.</p>
How to move model to double/float

username: rtkaratekid
cooked: <p>Pretty simple question. I’ve been scanning the docs and all I’ve found <a href="https://pytorch.org/cppdocs/api/structtorch_1_1jit_1_1script_1_1_module.html#_CPPv4N5torch3jit6script6Module2toEN2at6DeviceEN2at10ScalarTypeEb" rel="nofollow noopener">is this</a>. The problem I have is that all my searching hasn’t yielded how to pass at::ScalarType to that function or even what that even is.</p>
<p>I’ve tried passing “double,” a double variable, doing .double(), initializing an instance of at::ScalarType, and so on and just haven’t figure it out. I need to set to double to match how I’ve trained the model in my python script and because I enjoy the higher precision.</p>
<p>If you want code examples I can post some, but I think the issue is clear enough?<br>
Thanks!</p>

username: yf225
cooked: <p><a class="mention" href="/u/rtkaratekid">@rtkaratekid</a> Does <code>module-&gt;to(torch::kDouble)</code> work? Agreed that we need to make the docs better, and we plan to work on it in following months.</p>

username: rtkaratekid
cooked: <p><a class="mention" href="/u/yf225">@yf225</a> I really appreciate the reply!</p>
<p><code>module-&gt;to(torch::kDouble)</code> actually didn’t work, but…<br>
<code>module.to(torch::kDouble)</code> does.!</p>
<p>I’m just assuming that this is because the method <code>.to()</code> isn’t a pointer to a method/function?<br>
Anyway, I’m glad this was an easy fix! I just wish I had known that <code>torch::kDouble</code> is an <code>at::ScalarType</code>.</p>
<p>Also, just wanted to say that I think the docs are quite good, just maybe not 100% complete. But the forums here more than make up for it. I’ve enjoyed the supportive community.</p>

username: yf225
cooked: <p>Glad that I could be of help! Yes I think <code>.to()</code> instead of <code>-&gt;to()</code> works because <code>module</code> is an object of a subclass of <code>torch::nn::Module</code>. If <code>module</code> were a shared_ptr to a <code>torch::nn::Module</code> subclass object, we would have to use <code>-&gt;to()</code>.</p>
<p>We will be making a lot of improvements to the docs (the goal is to achieve parity with Python API docs, and also easy mapping between Python API and C++ API). Please stay tuned and feel free to ask any questions here in the forum. <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
Torch.quantization on JIT exported model

username: austinmw
cooked: <p>I saw the following example in the torch.quantization announcement:</p>
<blockquote>
<p>model = ResNet50()<br>
model.load_state_dict(torch.load(“model.pt”))</p>
<p>qmodel = quantization.prepare(<br>
model, {“”: quantization.default_qconfig})</p>
<p>qmodel.eval()<br>
for batch, target in data_loader:<br>
model(batch)</p>
<p>qmodel = quantization.convert(qmodel)</p>
</blockquote>
<p>Is there a way to get an example like this to work with a v1.3.0 JIT traced/exported model?</p>
Torch version problem

username: zahra
cooked: <p>Hi,</p>
<p>I want to install matchzoo-py that is based on pytorch but I receive this error:</p>
<pre><code class="lang-auto">Could not find a version that satisfies the requirement torch&gt;=1.2
chzoo-py) (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)
No matching distribution found for torch&gt;=1.2.0 (from matchzoo-py)
</code></pre>
<p>I upgrade the version by the following command:</p>
<pre><code class="lang-auto">pip install https://download.pytorch.org/whl/cu90/torch-1.2.0-cp36-cp36m-win_amd64.whl
pip install torchvision
</code></pre>
<p>after that I even could not import torch and I receive an error:</p>
<pre><code class="lang-auto">from torch._C import * (ImportError:...)
</code></pre>
<p>I would appreciate it if you guide me.</p>
<p>Thanks</p>

username: LeviViana
cooked: <p>Could you please report what is the output of <code>pip -V</code>?</p>

username: zahra
cooked: <p>Thanks for your reply.<br>
Of course, it is 19.0.3.<br>
Also, python version is 3.6.</p>

username: zahra
cooked: <p>I removed torch and then install torch by the following command:</p>
<pre><code class="lang-auto">pip3 install torch==1.2.0 torchvision==0.4.0 -f https://download.pytorch.org/whl/torch_stable.html
</code></pre>
<p>Now, I could import torch. Also, I could install matchzoo-py successfully but when I import torch and then import matchzoo, I receive this error that is related to pandas version I think:</p>
<pre><code class="lang-auto">AttributeError: module 'pandas' has no attribute 'core'
</code></pre>
<p>I would appreciate it if guide me.</p>
<p>Thansk</p>

username: zahra
cooked: <p>I removed pandas and installed version 0.24.2 and now I do not have any problems.<br>
For who read my question, suppose that this version is compatible with matchzoo-py(the package that I want to install). So, every other versions could be helpful for your application.</p>
<p>Many thanks</p>
No matching distribution found for torch

username: ati
cooked: <p>Hi<br>
i am going to install Bindsnet package with this command:</p>
<pre><code>pip install bindsnet -U
</code></pre>
<p>but when i run it , it get this error:</p>
<pre><code>ERROR: Could not find a version that satisfies the requirement torch&gt;=1.2.0 (from 
bindsnet) (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)

ERROR: No matching distribution found for torch&gt;=1.2.0 (from bindsnet)
</code></pre>
<p>I would be grateful if someone gives me a help</p>
100% accuracy in training data

username: Rohan_Kumar
cooked: <p>Is it possible to get 100% accuracy in training data, its relatively smaller data 3k data points, and was trained on GRU for 100 epochs. Is it ok (apart from overtraining) ? or is it not okay.</p>

username: ptrblck
cooked: <p>Overfitting would be the most critical part of perfectly fitting the training dataset. What other concerns are you considering?</p>
Most accurate neural network for image detection?

username: zimmer550
cooked: <p>Hi all. I am building a traffic counter and it requires detecting vehicles and people (drawing bounding box around them). So far I am using Yolov3 and getting above 90% accuracy for less congested areas and close to 90% accuracy even in highly congested urban areas. I was wondering if there is even a more accurate publicly available neural network? I know about Faster RCNN but so far, from what I have read, the most accurate neural network (as well as being fast) is Yolov3. Any suggestions will be appreciated. Thanks!</p>

username: Petrea_Tancau
cooked: <p>you can always do a fine-tuning of a pretrained model (like the faster-r-cnn of torchvision)</p>

username: samanmo
cooked: <p>actually yolov3 is not the most accurate net work, it trades accuracy with speed, so you could apply it also to live streams of videos, it is not even the fastest one while  yolo and yolov2 are faster and does not include detection in three layers, fast RCNN and SSD are still the most accurate ones.</p>

username: spanev
cooked: <p>Actually SSD offers the best accuracy-speed trade off and is still usable for real-time. In your case I would suggest to try it.<br>
You can find an accuracy and speed comparison <a href="https://medium.com/@jonathan_hui/object-detection-speed-and-accuracy-comparison-faster-r-cnn-r-fcn-ssd-and-yolo-5425656ae359" rel="nofollow noopener">here</a>.</p>
ONNX Usage in the browser

username: Petrea_Tancau
cooked: <p>I am trying to deploy a model in the browser for making predictions client side, once I export the model to onnx and I run in python everything works fine.</p>
<p>But the problem is when I run it in the browser, I always get the same prediction, and always get the following warning:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/6/c/6cf4264fcb14f201bd34ffb5ff4569ba6690b7cc.png" alt="image" data-base62-sha1="fxQEBWsFJ68nPACBqp1BsQhOHvu" width="283" height="69"></p>
<p>This is the input that causes the warning:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/e/0/e04bde9c16fccb681508bb37106e241712af61eb.png" alt="image" data-base62-sha1="w0dvceBj53PHQJz4ZURLpNjWTRV" width="637" height="169"><br>
As we can see, the dims are according to the model export input (1,3,224,224). (It is a fine-tuned resnet network).</p>
<p>Actually, I have a bounding box which I crop using the canvas api of javascript (new canvas, resize to 224,224 and getImageData on the new canvas), saving the image from the UI seems okay… I don’t get why it doesn’t work as expected, the prediction gives exactly the same values independently of the input photo (this does not happen in python).</p>
<p>Thank you for the attention, I am sure that many of you will face the requests of deploying models on mobile/browser.</p>

username: Petrea_Tancau
cooked: <p>UPDATE: The model runs succesfully with CPU BACKEND (WASM).</p>
<p>I am not sure why but my intuition is that this happens because the model was on cpu when exported</p>
Multidimensional List of Tensors to Multidimensional Tensor

username: Knievel
cooked: <p>I’m looking for a way to transform a multidimensional list of tensors that I assemble during one forward pass into a single tensor for further computations.</p>
<p>In three loops (over the direction (D), height (H) and width (W) of an image) I compute hidden state tensors of shape (Batch (B) x Hidden Units (H)), add them to a list, and at the end of this loop add this list to another list in the outside loop. In the end I have a list of shape (D x H x W) containing the (B x H) tensors.</p>
<p>How do I efficiently convert this to one tensor of shape (D x H x W x B x H) so I can apply the next weight matrix to it? Or is there a better way to collect the states in the first place? I thought about just concatenating the states together with torch.cat, but then I would need one initial tensor and always append to different dimensions.</p>
<p>(Also if I’m at it, what would be the way to combine the first and the last dimension of the result? Say I have 4 directions and hidden size of 16, how do I get (H x W x B x 64) from (4 x H x W x B x 16)? Just torch.view?)</p>
List of tensor elements to nubers

username: Deeply
cooked: <p>Just need to convert this list of tensors to an ordinary list.<br>
From:<br>
<code>[tensor([33]), tensor([7]), tensor([8]), tensor([11]), tensor([17]), tensor([20]), tensor([23]), tensor([24]), tensor([25]), tensor([26])]</code></p>
<p>To:<br>
<code>x = [33, 7, 8,..., 26]</code></p>

username: Tony-Y
cooked: <pre><code class="lang-auto">import torch
x = [torch.LongTensor([i]) for i in [33, 7, 8, 11, 17, 20, 23, 24, 25, 26] ]
torch.cat(x).tolist()
</code></pre>
Loading a torchscript model in C++: Cannot initialize CUDA without ATen_cuda library

username: spiovesan
cooked: <p>I have built a CNN with Pytorch with GPU support. It works fine in python and I would like to import it in my C++ application using VS2017. I am following these instructions <a href="https://pytorch.org/tutorials/advanced/cpp_export.html" rel="nofollow noopener">here</a> and <a href="https://medium.com/@boonboontongbuasirilai/building-pytorch-c-integration-libtorch-with-ms-visual-studio-2017-44281f9921ea" rel="nofollow noopener">here</a>.<br>
I can build and run the application, but I get this error when loading the model:</p>
<pre><code class="lang-auto">
</code></pre>
<p>error loading the model<br>
Cannot initialize CUDA without ATen_cuda library. PyTorch splits its backend into two shared<br>
libraries: a CPU library and a CUDA library; this error has occurred because you are trying to use<br>
some CUDA functionality, but the CUDA library has not been loaded by the dynamic linker for some<br>
reason.  The CUDA library MUST be loaded, EVEN IF you don’t directly use any symbols from the CUDA<br>
library! One common culprit is a lack of -Wl,–no-as-needed in your link arguments; many dynamic<br>
linkers will delete dynamic library dependencies if you don’t depend on any of their symbols.  You<br>
can check if this has occurred by using ldd on your binary to see if there is a dependency on<br>
*_cuda.so library. (initCUDA at<br>
C:\w\1\s\windows\pytorch\aten\src\ATen/detail/CUDAHooksInterface.h:63)<br>
(no backtrace available)</p>
<pre><code class="lang-auto"></code></pre>
<p>How can I fix with VS2017 and CUDA 10?</p>

username: tom
cooked: <p>But you did get a cuda-enabled libtorch?<br>
You could try calling <code>cuda_lazy_init</code> before if you did.</p>
<p>Best regards</p>
<p>Thomas</p>

username: spiovesan
cooked: <p>I solved using the nightly <a href="https://download.pytorch.org/libtorch/nightly/cu100/libtorch-win-shared-with-deps-latest.zip" rel="nofollow noopener">build</a>, that ships  <code>libtorch</code>  with all the needed  <code>cuda</code>  dlls and the correct conda python 3.7 pytorch 1.3 (the conda installer for <a href="http://conda%20install%20pytorch%20torchvision%20cudatoolkit%3D10.0%20-c%20pytorch/" rel="nofollow noopener">Windows</a> does not install the latest  <code>pytorch stable 1.2</code> but an older  <code>1.0.1</code>) .</p>

username: tom
cooked: <p>Supergood, thank you for reporting back!</p>
<p>Best regards</p>
<p>Thomas</p>
Automatic differentiation with custom layer

username: vikvik
cooked: <p>Dear PyTorch users,</p>
<p>I am relatively new to PyTorch and trying to implement a custom layer.</p>
<ol>
<li>My forward() has a set of non-trivial set of operations and I am wondering whether the gradient will be automatically computed or do I have to implement a new function that returns these gradients, (through backward())?</li>
</ol>
<p>Here is my custom layer,</p>
<pre><code class="lang-auto">class SNE(nn.module):
    
    def __init__(self, input_features, output_features, kmeans):
        
        super(SNE, self).__init__()
        self.input_features = input_features
        self.output_features = output_features

        # Register Parameters
        self.weight = nn.Parameter(torch.Tensor(output_features, input_features))

        # Initialize wts from the kmeans algorithm
        self.weight.data = kmeans

    def forward(self, input):

        output = 1 + torch.cdist(input, self.weight) # Compute cdist
        output = 1/ output # Invert
        

        for j in np.arange(output.size()[0]):
            output[j] = output[j]/output[j].sum() #Normalize
        
     return output
</code></pre>
<p>An alternative would be just call a <strong>new_func()</strong> (inherited from Function) in the <strong>forward()</strong> above with a forward() and backward() implementation. But then, my backward() will look a bit messy and likely to be inefficient or error prone.</p>
<ol start="2">
<li>My <strong>init</strong>() is taken directly from the template given <a href="https://pytorch.org/docs/stable/notes/extending.html" rel="nofollow noopener">HERE</a>. Also, I do not really understand what is going on with <strong>output_features</strong> in this template, would someone minding explaining me this?</li>
</ol>
<p>Thanks, in advance.</p>
Torch.matmul error

username: vainaijr
cooked: <pre><code class="lang-auto">torch.matmul(torch.randn(16, 4, 7056, 10), torch.randn(16, 4, 10, 7056))
</code></pre>
<p>gives error</p>
<pre><code class="lang-auto">RuntimeError: CUDA out of memory. Tried to allocate 11.87 GiB (GPU 0; 10.91 GiB total capacity; 99.79 MiB already allocated; 9.00 GiB free; 4.21 MiB cached)
</code></pre>
<p>any alternative?</p>

username: Tony-Y
cooked: <p>The output size is</p>
<pre><code class="lang-auto">torch.Size([16, 4, 7056, 7056])
</code></pre>
<p>So, it really requires 11.87 GB on GPU. If you want whole output at once, you need a GPU with more memory than 12 GB.</p>

username: bnikolic
cooked: <p>You don’t give much of a context, maybe this is obvious but in case not:</p>
<p>You can allocate the matrices on the CPU, then iterate over the first two dimensions, and send the sliced tensors to GPU for the actual <code>matmult</code>  (which only computes on the last two dimensions).</p>
Automatically determine max. batch-size for inference

username: Haydnspass
cooked: <p>Hey,</p>
<p>I want to use the max. batch-size possible for inference.<br>
I thought of reading out CUDA’s free memory and by knowledge of my tensor 's physical size calculating the batch_size. Is this reasonable, or are there more sophisticated methods to do this?</p>
<p>What would be the proper function to call for determining the free space available for my images?<br>
I guess reading out the currently used memory could be done via</p>
<pre><code class="lang-auto">#include &lt;c10/cuda/CUDACachingAllocator.h&gt; 
[...]
uint64_t curr_mem = c10::cuda::CUDACachingAllocator::currentMemoryAllocated(0);
</code></pre>
<p>But I’d also need the total memory …</p>
<p>Cheers,<br>
Lucas</p>

username: tom
cooked: <p>I don’t think PyTorch provides this, but you could try your luck with <a href="https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__MEMORY.html#group__CUDART__MEMORY_1g376b97f5ab20321ca46f7cfa9511b978" rel="nofollow noopener">cudaMemGetInfo</a>.</p>
<p>Best regards</p>
<p>Thomas</p>
Where is the gpu implementation of nonzero kernel

username: ScXfjiang
cooked: <p>Where is the gpu implementation of nonzero kernel, I can’t find it, thanks!</p>

username: tom
cooked: <p>It’s in THC (legacy torch) as <a href="https://github.com/pytorch/pytorch/blob/91959aa3d3a55c358630c9123810877013294cb2/aten/src/THC/generic/THCTensorMath.cu#L245" rel="nofollow noopener">nonzero in /aten/src/THC/generic/THCTensorMath.cu</a>.</p>
<p>Best regards</p>
<p>Thomas</p>