akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Libtorch load os cpu/gpu

username: airwater
cooked: <p>I’m doing a project, deploy the model under linux to windows.<br>
About Project migration, I want to ask  a question.<br>
Call libtorch under windows, what are the requirements for the operating system, libtorch version?<br>
For example:<br>
Can I load the trained model under Linux to Windows?<br>
Can be between different pytorch and libtorch， workd it ?</p>
Pytorch build from source with onnx enable

username: jinfagang
cooked: <p>How to enable onnx python support when building pytorch from source.</p>
<p>The default config disabled onnx python, and I can not import onnx from torch.</p>
<p>Anybody could help??</p>
<pre><code class="lang-auto">python3 setup.py install
</code></pre>
<p>This is currently how I know to build pytorch from source.</p>

username: LeviViana
cooked: <p>onnx is a submodule of PyTorch’s source code, normally it is enabled by default. What is printed when you enter the following snippet ?</p>
<pre><code class="lang-python">import torch
torch.onnx.__path__
</code></pre>
Analyzing Hook drastically decreases performance

username: FruitVinegar
cooked: <p>Hi. I’m using a PyTorch Hook to analyzing intermediate feature map’s statistics with a large-scale model training.<br>
Also I’m using multi-GPU to accelerate the training process.(with torch.nn.DataParallel)</p>
<p>The problem is, GPU utilization is drastically drops down when my Hook is executing(I compared GPU utilization with when I don’t activated my Hook - naive training without analyzing)</p>
<p>I know this is Indispensable situation because my analyzing Hook is quite high-cost function, but all my multi-GPU are freezing in Hook phase, which seems not reasonable.</p>
<p>I think there’s some solution, such as Hook-specify GPU setting, like:<br>
GPU 0~3: model training<br>
GPU 4: gather intermediate feature map and analyze</p>
<p>Is it possible? Or is there any good solution to solve my problem?<br>
Any answers will be welcomed. Thanks.</p>

username: albanD
cooked: <p>Hi,</p>
<p>Hooks are very slow for multiple reasons: They are python code and cannot run concurrently (due to python limitations). They require the value of the Tensor usually and so will for the GPU to sync all computation (wait for everything to run, execute, then start queuing more work). Since multi-gpu splits a batch between the gpus, the whole batch runs as slow as the slowest part of the batch. In your case, that would be the GPU that has the hooks.</p>
<p>A possible solution could be to only collect these statistics every 10 batch?</p>

username: FruitVinegar
cooked: <p>Sorry for late appreciation.<br>
I totally re-wrote my code and changed my algorithm, so I can reduce 11 Days -&gt; 4 Days training time with my hooks.<br>
But oscillating GPU Utilization, such as:</p>
<p>almost 100% GPU util(training process: like backpropagation)<br>
-&gt; 4~10% GPU util(my hooks; even I wrote all operations to be executed in GPU)<br>
-&gt; 100%…<br>
(repeating)</p>
<p>This speed-difference between optimized training process and my custom task is still bottleneck.<br>
I hope pytorch to have more powerful optimizing semantics on hooks, cause it’s very useful tool for researchers.</p>
<p>Thanks again for your answers.</p>
How to improve conditional tensor operation?

username: Nick_8229
cooked: <p>I have tensor condition operation, such as</p>
<pre><code class="lang-auto">for i:
   if check(rate[i]):
      rate[i] = reset(rate[i])
</code></pre>
<p>right now, i implemented as following,<br>
<code>rate = torch.where(check(rate), reset(rate), rate)</code></p>
<p>It turns out that even for tensor where check(rate) is false, reset(rate) is still operated, kind of like</p>
<pre><code class="lang-auto">new_rate = reset(rate)
rate = torch.where(check(rate), new_rate, rate)
</code></pre>
<p>I am wondering anyway i could improve the performance here, if reset(rate) is really expensive.</p>

username: Nick_8229
cooked: <p>It seems mask should help the case, like</p>
<pre><code class="lang-auto">mask = check(rate)
rate[mask] = reset(rate[mask])
</code></pre>
<p>And now only the tensor of the mask will be operated. However, it is even slower.</p>

username: Nick_8229
cooked: <p>Here is an example,</p>
<pre><code class="lang-auto">import torch
import time
import copy

def reset(t):
   return 1.0 / torch.log(1.0 + t)

source1 = torch.FloatTensor(10000, 10000)
source1.uniform_()
source2 = copy.deepcopy(source1)

start = time.time()
mask = source1 &lt; 0.5
source1[mask] = reset(source1[mask])
print('test1 takes: ', time.time() - start)

start = time.time()
mask = source2 &lt; 0.5
source2 = torch.where(mask, reset(source2), source2)
print('test2 takes: ', time.time() - start)
</code></pre>
<p>On my machine,<br>
test1 takes:  3.4766557216644287<br>
test2 takes:  1.7760250568389893</p>
C++ shared_ptr<torch::jit::script::Module>

username: airwater
cooked: <p>I meet a problem, it’s blocking me for a week. If you have a solution, please tell me, or welcome to discuss with together. Thank to you !</p>
<p>Problems:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/d/dc2a1d70916c1edeadd787e2e45463af66ddf8f1.png" data-download-href="https://discuss.pytorch.org/uploads/default/dc2a1d70916c1edeadd787e2e45463af66ddf8f1" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/d/dc2a1d70916c1edeadd787e2e45463af66ddf8f1_2_690x30.png" alt="image" data-base62-sha1="vpFh13mpSs6E5m928WAYVJUNSHn" width="690" height="30" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/d/dc2a1d70916c1edeadd787e2e45463af66ddf8f1_2_690x30.png, https://discuss.pytorch.org/uploads/default/optimized/2X/d/dc2a1d70916c1edeadd787e2e45463af66ddf8f1_2_1035x45.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/d/dc2a1d70916c1edeadd787e2e45463af66ddf8f1.png 2x" data-dominant-color="2A2A29"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1058×47 9.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>vs smart Tip: There is no user-defined appropriate conversion from torch::jit::script::Module to std::shared_ptr<a>torch::jit::script::Module</a></p>

username: yf225
cooked: <p>Please see:</p><aside class="quote" data-post="3" data-topic="53462">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yf225/48/3418_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/broken-api-in-libtorch-1-2/53462/3">Broken API in libtorch 1.2</a> <a class="badge-category__wrapper " href="/c/c/11"><span data-category-id="11" data-drop-close="true" class="badge-category " title="Topics related to the C++ Frontend, C++ API or C++ Extensions"><span class="badge-category__name">C++</span></span></a>
  </div>
  <blockquote>
    Yes the PyTorch C++ API is still considered “beta” stability. Specifically, we changed torch::jit::load(s_model_name) to return torch::jit::script::Module instead of std::shared_ptr&lt;torch::jit::script::Module&gt; because torch::jit::script::Module is now a reference type, and this change is documented in the release note ("[jit] script::Module is now a reference type" in <a href="https://github.com/pytorch/pytorch/releases/tag/v1.2.0" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/releases/tag/v1.2.0</a>).
  </blockquote>
</aside>


username: airwater
cooked: <p>Thank you very much. Nothing is more enjoyable than getting a solution. Thanks again!</p>
Maximum filter activation

username: rle1323
cooked: <p>The keras team made a pretty good blog post about producing images that maximally activate each of the filters in a CNN, for the purpose of visualizing the features that a CNN learns. It’s linked below. Has anyone seen or tried any implementations of this using PyTorch tools?</p>
<p><a href="https://blog.keras.io/how-convolutional-neural-networks-see-the-world.html" class="onebox" target="_blank" rel="nofollow noopener">https://blog.keras.io/how-convolutional-neural-networks-see-the-world.html</a></p>
How to freeze a network during training?

username: John1231983
cooked: <p>I have two networks: net1 and net2 and an input x. I want to feed the input <code>x</code> to the net1 to generate the <code>pred x1</code>. Then, the <code>pred x1</code> is fed to the network 2 to generate <code>pred x2</code>.</p>
<p>I want to freeze network <code>net1</code>, while train the <code>net2</code>. The loss is computed as the mse between <code>pred x1</code> and <code>pred x2</code>. I shows my implementation but I checked that the weights of net1 is still updated during training. How can I fix it?</p>
<pre><code class="lang-auto">net1=Model_net1()
net1.eval()
net2=Model_net2()
net2.train()
for param in net1.parameters():
            param.requires_grad = False
with torch.no_grad():  
    pred_x1= net1(x)
    pred_x2= net2(pred_x1)
    loss = mse(pred_x1, pred_x2)
    loss.backward()
    optimizer.step()
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/a/ace75e8e2c42ad85ff59f0c4ff7625e411bc7562.png" data-download-href="https://discuss.pytorch.org/uploads/default/ace75e8e2c42ad85ff59f0c4ff7625e411bc7562" title="Screenshot%20from%202019-09-05%2015-38-33"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/a/ace75e8e2c42ad85ff59f0c4ff7625e411bc7562_2_690x176.png" alt="Screenshot%20from%202019-09-05%2015-38-33" data-base62-sha1="oFzPycx8pNGW30hk9C75w9A5wPg" width="690" height="176" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/a/ace75e8e2c42ad85ff59f0c4ff7625e411bc7562_2_690x176.png, https://discuss.pytorch.org/uploads/default/optimized/2X/a/ace75e8e2c42ad85ff59f0c4ff7625e411bc7562_2_1035x264.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/a/ace75e8e2c42ad85ff59f0c4ff7625e411bc7562.png 2x" data-dominant-color="F9F8F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot%20from%202019-09-05%2015-38-33</span><span class="informations">1053×270 11.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>Could you try to detach <code>pred_x1</code> before passing it to <code>net2</code>?<br>
Also, did you pass the parameters of <code>net1</code> to the optimizer and are you using weight decay?</p>

username: John1231983
cooked: <p>Yes. I used the detach() also and did not use any optimize for net1</p>

username: ptrblck
cooked: <p>How did you check that <code>net1</code>'s parameters were updated?</p>

username: John1231983
cooked: <p>Sorry. I mistaken. I have check it and the parameter is not changed. Some bug in my code. Sorry bro</p>
Understanding code organization: where is `log_softmax` really implemented?

username: cwindolf
cooked: <p>Hi –</p>
<p>So, I’m new to PyTorch, and I’m spending a lot of time in the docs. Recently, I was digging around trying to find out how <code>log_softmax</code> is implemented.</p>
<p>I started out looking at the source for <a href="https://pytorch.org/docs/stable/nn.html#logsoftmax" rel="nofollow noopener">torch.nn.LogSoftmax</a>, which is implemented with <a href="https://pytorch.org/docs/stable/nn.functional.html#log-softmax" rel="nofollow noopener">torch.nn.functional.log_softmax</a>. OK, so I went to the docs for that and clicked the source link, and found that this function is implemented by using the method <code>.log_softmax</code> of its input.</p>
<p>I figured that this meant that <code>.log_softmax</code> would probably be a method of <code>torch.Tensor</code>, so I went to the docs for <code>Tensor</code>, but was unable to find any such method. I’m wondering, where is this method implemented?</p>
<p>I’d also appreciate any general insight into how <code>torch</code>'s organization influences the location of this code/other code like it.</p>
<p>Thank you!</p>

username: ptrblck
cooked: <p>You’ll find the CPU implementation <a href="https://github.com/pytorch/pytorch/blob/420b37f3c67950ed93cd8aa7a12e673fcfc5567b/aten/src/ATen/native/SoftMax.cpp#L146" rel="nofollow noopener">here</a>.</p>
<p>PS: <code>rgrep</code> is often a useful tool to find the corresponding function.<br>
The GitHub search function is often also quite helpful.</p>
Using MaxUnpool for TransferLearning

username: DidierDeschamps
cooked: <p>Hi,</p>
<p>I have a question about the use of the MaxUnpool with Pytorch.</p>
<p>I Made an Autoencoder (AE) symetric with 5/5 layers:<br>
Encoder: [ConvLayer+MaxPool]*5<br>
Decoder: [ConvLayer+MaxUnpool]*5</p>
<p>At the end of the encoder I achieved a reduction of 3%. Astonishingly my AE is performing perfectly. I almost have an overlap of inputs/ouputs.</p>
<p>It seems like the Maxunpool is a way of cheating in the learning. Keeping the indexes sounds too powerful. I believe that the Network is not learning enough and expect that the MaxUnpools are enough to reconstruct the data backward.</p>
<p>The final goal of my project is to perform transfer learning using the Encoder that should learn and understand the “main features” of my dataset.</p>
<p>Do you think that the MaxUnpool are too powerful and are not a good solution for what I am aiming?</p>

username: ptrblck
cooked: <p>I’m not sure if <code>MaxUnpool</code> is too powerful, as the majority of the activation output should be zero, so that your model still would have to learn the other values.<br>
Did you try to use the encoder for your other use case?<br>
If so, did it perform poorly?</p>
<p>Just as a test you could try to use transposed convolutions and see, if that helps your model (especially the encoder as it seems to be the main use case).</p>

username: DidierDeschamps
cooked: <p>Thank you so much for your answer. We haven’t tried yet on the other models, but we think we are going to move forward and check the performance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> !</p>

username: DidierDeschamps
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> We finally did a test that was confirming that MaxUnpool was too powerful.<br>
We fed the encoder with an image to get the indices and then put a random vector before the decoder.</p>
<pre><code class="lang-auto">def forward(self, input):
    latent_vector, indices = self.encoder(input)
    # Here we overwrite the latent vector with a random one
    latent_vector = torch.randn(self.dim_latent_vector) 
    output = self.decoder(latent_vector, indices)
    return output
</code></pre>
<p>The decoder performed a successful reconstitution of the input expected anyway.<br>
The following <a href="https://orbit.dtu.dk/ws/files/128168371/double.pdf" rel="nofollow noopener">paper</a> details the phenomena encountered.</p>

username: ptrblck
cooked: <p>Thanks for the follow-up!<br>
That’s an interesting observation. What did you end up using?</p>

username: DidierDeschamps
cooked: <p>For now we are trying a fully CNN model but it’s learning slowly so we don’t have any satisfactory results yet <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> .</p>
Why Tensor shape is not a Tensor?

username: dashesy
cooked: <p>Why is <code>tensor.shape</code>/<code>tensor.size()</code> not a tenosr? what is the need to also have a <code>tensor.Size</code>?<br>
When tracing <code>tensor.shape[0]</code> is a tensor but when not tracing <code>tensor.shape[0]</code> is an int. So I have to do check with <code>isinstance</code> like this:</p>
<pre><code class="lang-python">@torch.jit.script
def center_slice_helper(x, h_offset, w_offset, h_end, w_end):
    return x[:, :, h_offset:h_end, w_offset:w_end]


class CenterCrop(nn.Module):
    def __init__(self, crop_size):
        """Crop from the center of a 4d tensor
        Input shape can be dynamic
        :param crop_size: the center crop size
        """
        super(CenterCrop, self).__init__()
        self.crop_size = crop_size

    def extra_repr(self):
        """Extra information
        """
        return 'crop_size={}'.format(
            self.crop_size
        )

    def forward(self, x):
        h_offset = (x.shape[2] - self.crop_size) / 2
        w_offset = (x.shape[3] - self.crop_size) / 2
        if not isinstance(h_offset, torch.Tensor):
            h_offset, w_offset = torch.tensor(h_offset), torch.tensor(w_offset)
        h_end = h_offset + self.crop_size
        w_end = w_offset + self.crop_size
        return center_slice_helper(x, h_offset, w_offset, h_end, w_end)
</code></pre>
<p>If <code>tensor.shape</code> was a tensor we could use it like other tensors in the network</p>
Subclassing nn.ModuleDict vs nn.Module

username: vainaijr
cooked: <p>When should one subclass nn.ModuleDict over nn.Module?</p>
<p>for example, here,<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/civodlu/trw/blob/42bb09e4bee07c85c3d9585b07e0e23f7a60906f/tutorials/dcgan_mnist.py" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/civodlu/trw/blob/42bb09e4bee07c85c3d9585b07e0e23f7a60906f/tutorials/dcgan_mnist.py" target="_blank" rel="nofollow noopener">civodlu/trw/blob/42bb09e4bee07c85c3d9585b07e0e23f7a60906f/tutorials/dcgan_mnist.py</a></h4>
<pre><code class="lang-py">import trw.train
import trw.datasets
import torch
import torch.nn as nn
import torch.nn.functional as F
import functools
import collections

batch_size = 32
latent_size = 64
mnist_size = 28 * 28
hidden_size = 256


class Flatten(nn.Module):
    def forward(self, i):
        return i.view(i.shape[0], -1)


class Generator(nn.Module):
</code></pre>

  This file has been truncated. <a href="https://github.com/civodlu/trw/blob/42bb09e4bee07c85c3d9585b07e0e23f7a60906f/tutorials/dcgan_mnist.py" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>is it better to use nn.ModuleDict whenever merging multiple neural networks?</p>
<p>so,</p>
<pre><code class="lang-auto">class Y(nn.ModuleDict):
  def __init__(self):
    super().__init__()
    self['NetA'] = NetA()
    self['NetB'] = NetB()
</code></pre>
<pre><code class="lang-auto">def X(nn.Module):
  def __init__(self):
    super().__init__()
    self.modelone = NetA()
    self.modeltwo = NetB()
</code></pre>
<p>which is the preferred way?</p>

username: ptrblck
cooked: <p>It might depend on your use case, but <code>nn.ModuleDict</code> is just a container (<code>dict</code>), which stores modules and is used to register these modules properly inside a parent <code>nn.Module</code>.</p>
<p>Based on your code snippet, I would derive from <code>nn.Module</code> (your <code>X</code> class).</p>
What is the correct way of using trchvision.utils.make_grid()

username: Shisho_Sama
cooked: <p>Hello all, I have been trying to use this method but fail each time. basically I have two images that I stored in a list (using img_lst.append((img1,img2)). one image is the input image and the other is its reconstruction.<br>
They are both gray scale images.<br>
This is how I tried to do :</p>
<pre><code class="lang-auto">def visualize(imgs_list, rows=2, cols=10):
    fig = plt.figure(figsize=(10,5))
    print(f'number of samples: {len(imgs_list)}')
    for i in range(len(imgs_list)):
        img,recons = imgs_list[i]
        img = img.cpu().permute(1,2,0)
        recons = recons.cpu().detach().permute(1,2,0)
        f = [img,recons]
        ax = fig.add_subplot(rows, cols, i+1, xticks=[], yticks=[])
        ax.imshow(torchvision.utils.make_grid(f), cmap='Greys_r')
</code></pre>
<p>but this fails with the error :</p>
<blockquote>
<p>RuntimeError: The expanded size of the tensor (3) must match the existing size (28) at non-singleton dimension 0.  Target sizes: [3, 28, 1].  Tensor sizes: [28, 28, 1]</p>
</blockquote>
<p>if I remove permute, it complains about the dimensions being invalid :</p>
<blockquote>
<p><strong>TypeError</strong> : Invalid dimensions for image data</p>
</blockquote>
<p>I’m completely lost here! how is this supposed to work?<br>
Can anyone kindly please help me understand this?<br>
Thanks a lot in advance</p>

username: spanev
cooked: <p>Hi <a class="mention" href="/u/shisho_sama">@Shisho_Sama</a></p>
<p>The images have to be in the CHW layout (channel first), with <code>img.shape == recons.shape</code>.<br>
Can you check the shapes by printing them?</p>

username: Shisho_Sama
cooked: <p>They are in CHW. and here is the print output for<br>
<code>print(img.shape,recons.shape)</code><br>
<code>torch.Size([1, 28, 28]) torch.Size([1, 28, 28])</code></p>

username: spanev
cooked: <p>Are you sure that the error comes from <code>make_grid</code>? It might come from <code>imshow</code> as well. Can you try to run it without <code>ax.imshow</code>?</p>

username: Shisho_Sama
cooked: <p>You were right! The error was caused by the matplotlibs imshow().<br>
I had to do sth like this :</p>
<pre><code class="lang-auto">        x = torchvision.utils.make_grid([img,recons])
        ax.imshow(x.numpy().transpose(1,2,0))
</code></pre>

username: spanev
cooked: <p>Oh right, it just came to my mind, matlpotlib’s imshow expects HWC images!<br>
I’m glad it worked <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=9" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:"></p>
Evaluator of pedestrian detection model

username: ahmed
cooked: <p>Does anyone know of an existing evaluator? I know that the <code>CocoEvaluator</code> was used in the pytorch tutorial: <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html</a>, but does anyone know of any others?</p>
<p>I would like to evaluate a model for pedestrian detection and I have seen in many papers that <code>miss rate per false positives per image</code> is often used. Due to lack of time, it would be better if I could find an existing evaluator, rather than write from scratch.</p>
Output variable creation in depthwise convolution

username: Ahmad_Khan
cooked: <p>Where should I change to create a new argument similar to output which returns a tensor in depth wise convolution?</p>

username: albanD
cooked: <p>Hi,</p>
<p>I’m not sure to understand your question.<br>
You want to create a Tensor that is the same size as the output of a given convolution?<br>
Could you elaborate on why you want to be able to do that?</p>

username: Ahmad_Khan
cooked: <p>Thanks for replying ,yeah that’s right I would like to create a tensor of same size as output .</p>

username: albanD
cooked: <p>There is no existing method to do this. But you can use the formula from the <a href="https://pytorch.org/docs/stable/nn.html#convolution-layers" rel="nofollow noopener">convolution doc</a> where the output shape is specified as a function of the parameters. Then create a new Tensor of that size.</p>

username: Ahmad_Khan
cooked: <p>I would like to do that in C source code in SpatialDepthwiseConv.cpp in Aten<br>
How can I do that.</p>

username: albanD
cooked: <p>Hi,</p>
<p>Why would you want that in the cpp code? It will be much simpler to do it in a python function in your code no?</p>
Why is GraphFuser a NonDiffOptimization?

username: X.Guo
cooked: <p>In <a href="https://github.com/pytorch/pytorch/blob/8554416a199c4cec01c60c7015d8301d2bb39b64/torch/csrc/jit/graph_executor.cpp#L548" rel="nofollow noopener">graph_executor.cpp</a>,<br>
If the graph to be optimized needs Gradient, <code>runNondiffOptimization(gradient.f)</code> will be called.</p>
<p>runNondiffOptimization() runs some optimizations including BatchMM and FuseGraph.<br>
I wonder why are these optimization non-differentiable?</p>
<p>Take the <code>FuseGraph</code> as an example. It fuses continuous point-wise ops.<br>
If it can fuse <code>f(g(x))</code>, why can’t it fuse <code>df(g(x)) * dg(x)</code>.</p>
<p>Or it’s simply because the current implementation of FuseGraph does not support this?</p>
Dropout: non-deterministic on different GPUs

username: matthew_zeng
cooked: <p>The simple code below yields different results on different GPUs (e.g., 1070 got 998874 but 1080Ti got 998836). I wonder if I did something wrong or it is just impossible to get the same result on different GPUs?</p>
<pre><code class="lang-python">import torch
import numpy as np 
import random 

seed = 0
torch.backends.cudnn.benchmark = False
torch.backends.cudnn.deterministic = True
torch.manual_seed(seed)
torch.cuda.manual_seed(seed)
torch.cuda.manual_seed_all(seed)  # if you are using multi-GPU.
np.random.seed(seed)  # Numpy module.
random.seed(seed)  # Python random module.


a = torch.ones(1000,1000).to('cuda:0')
dropout = torch.nn.Dropout(0.5).cuda()
b = dropout(a)

print(torch.sum(torch.abs(b)))
</code></pre>

username: ptrblck
cooked: <p>Are you using the same PyTorch version (CUDA, cudnn)?</p>
<p>Getting the same “random” numbers on different hardware is sometimes quite hard.<br>
However, using your code, I get the same result (<code>tensor(1000260., device='cuda:0')</code>) for:</p>
<ul>
<li>PyTorch 1.2.0, CUDA10.0.130, cudnn7602, TitanV</li>
<li>PyTorch master ~few weeks old, CUDA10.1.168, cudnn7601, V100</li>
</ul>

username: matthew_zeng
cooked: <p>Yes, the same environment. I don’t have a TitanV to try but I guess it is quite similar to V100 so they could yield the same result.</p>
<p>More tests:</p>
<p>My local server (PyTorch 1.20, CUDA 10.0.130, CUDNN7602, 2080Ti) got 998908.<br>
Instances on Google Cloud using the official pytorch 1.20 image (exactly same versions as above): got 1000260 on V100 but 999100 on K80.</p>

username: ptrblck
cooked: <p>Yeah, that might be the case. However, I would assume the <code>1070</code> and <code>1080Ti</code> are also similar to each other (same architecture).</p>

username: yezhe
cooked: <p>My K80 also got 999100. My two 1080s on quite different machines both got 998662.</p>

username: matthew_zeng
cooked: <p>I checked again, my 1070 and 1080ti machines have the same CUDA CUDNN PyTorch versions, but their results are different …<img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=9" title=":joy:" class="emoji" alt=":joy:"></p>
Organizing optimizer trainable params per epoch

username: Hovnatan_Karapetyan
cooked: <p>Hi, I have a network in which I want to train one set of params at odd numbered epochs and another set of params at even numbered epochs. What is the best way to organize this in PyTorch? Thanks!</p>

username: spanev
cooked: <p>Hi <a class="mention" href="/u/hovnatan_karapetyan">@Hovnatan_Karapetyan</a>,</p>
<p>Given P1, P2 the two param sets, I think the best way is to create two optimizers with the same hyperparameters:</p>
<pre><code class="lang-auto">optim_even = torch.optim.SGD(P1, lr=0.01)
optim_odd  = torch.optim.SGD(P2, lr=0.01)

for e in range(epochs):
    optim = optim_even if e % 2 == 0 else optim_odd
    for x in dataloader:
         #use optim normally here
</code></pre>
<p>There might be some further <em>optimizations</em> to do depending on the optimizer you use (SGD should be with his approach) and on your model topology, but this should be sufficient in simple use cases.</p>
How to run PyTorch code in parallel on several nodes?

username: JohnCwok
cooked: <p>Hi,</p>
<p>I am using a cluster to train a recurrent neural network. Since PyTorch seems to automatically thread, it lets me use all the cores of one machine in parallel without having to explicitly program for it. This is great !</p>
<p>Now when I try to use several nodes at the same time using a script like this one :</p>
<pre><code class="lang-auto">#$ -S /bin/bash
#$ -N comparison_of_architecture
#$ -pe mvapich2-rostam 32
#4 -tc 4
#$ -o /scratch04.local/cnelias/Deep-Jazz/logs/out_comparison_training.txt
#$ -e /scratch04.local/cnelias/Deep-Jazz/logs/err_comparison_training.txt
#$ -t 1
#$ -cwd
</code></pre>
<p>I see that 4 nodes are being used but only one is actually doing work, so “only” 32 cores are in use.</p>
<p>I have no knowledge of parallel programming and I don’t understand a thing in the tutorial provided on PyTorch’s website, I am afraid this is completely out of my scope.</p>
<p>Are you aware of a simple way to let a PyTorch program run on several machines without having to explicitly program the exchanges of the messages and computation between these machines ?</p>
<p>PS : I unfortunately don’t have a GPU and the cluster I am using also doesn’t, otherwise I would have tried it.</p>
ValueError: Target size (torch.Size([16])) must be the same as input size (torch.Size([16, 1]))

username: mnauf
cooked: <pre><code class="lang-auto">ValueError                                Traceback (most recent call last)
&lt;ipython-input-30-33821ccddf5f&gt; in &lt;module&gt;
     23         output = model(data)
     24         # calculate the batch loss
---&gt; 25         loss = criterion(output, target)
     26         # backward pass: compute gradient of the loss with respect to model parameters
     27         loss.backward()

C:\Users\mnauf\Anaconda3\envs\federated_learning\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
    487             result = self._slow_forward(*input, **kwargs)
    488         else:
--&gt; 489             result = self.forward(*input, **kwargs)
    490         for hook in self._forward_hooks.values():
    491             hook_result = hook(self, input, result)

C:\Users\mnauf\Anaconda3\envs\federated_learning\lib\site-packages\torch\nn\modules\loss.py in forward(self, input, target)
    593                                                   self.weight,
    594                                                   pos_weight=self.pos_weight,
--&gt; 595                                                   reduction=self.reduction)
    596 
    597 

C:\Users\mnauf\Anaconda3\envs\federated_learning\lib\site-packages\torch\nn\functional.py in binary_cross_entropy_with_logits(input, target, weight, size_average, reduce, reduction, pos_weight)
   2073 
   2074     if not (target.size() == input.size()):
-&gt; 2075         raise ValueError("Target size ({}) must be the same as input size ({})".format(target.size(), input.size()))
   2076 
   2077     return torch.binary_cross_entropy_with_logits(input, target, weight, pos_weight, reduction_enum)

ValueError: Target size (torch.Size([16])) must be the same as input size (torch.Size([16, 1]))
</code></pre>
<p>I am working on the Horses vs humans dataset. <a href="https://github.com/mnauf/horses_humans/blob/master/Untitled.ipynb" rel="nofollow noopener">This is my code</a>. I am using <code>criterion = nn.BCEWithLogitsLoss()</code> and <code>optimizer = optim.RMSprop(model.parameters(), lr=0.01</code>). My final layer is <code>self.fc2 = nn.Linear(512, 1)</code> with softmax activation function applying on it.</p>
<p><code>16</code> is the batch size. Since the error says <code>ValueError: Target size (torch.Size([16])) must be the same as input size (torch.Size([16, 1]))</code>. I don’t understand, where do I need to make change, to rectify the error.</p>

username: mnauf
cooked: <p><code>target = target.unsqueeze(1)</code>, before passing target to criterion, changed the target tensor size from <code>[16]</code> to <code>[16,1]</code>. Doing it solved the issue. Furthermore, I also needed to do <code>target = target.float()</code> before passing it to <code>criterion</code>, because our outputs are in float. Besides, there was another error in the code. I was using sigmoid activation function in the last layer, but I shouldn’t because the criterion I am using already comes with sigmoid builtin.</p>
Question about torch.distributions.normal

username: Xiaodong_MEI
cooked: <p>If the input are multi-dimensional tensors, dose it wil generate seperate normal distributions?<br>
For example,<br>
dis = Normal ( torch.tensor([1.0, 200.0]), torch.tensor([1.0, 2.0]) )</p>
<p>Will it give tow seperate normal distributions with Normal1(mu = 1.0, sigma = 1.0) and Normal2(mu = 200.0, sigma = 2.0) ?</p>
<p>Thanks for your answers</p>

username: phan_phan
cooked: <p>Yes it will exactly do that <img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=9" title=":+1:" class="emoji" alt=":+1:"></p>
Torch.jit.trace unexpected error with `torch.cat(..., dim=-1)`

username: imaluengo
cooked: <p>Find below a Minimum Reproducible Example that crashes both in Pytorch 1.1 and Pytorch 1.2 with CUDA (it works with CPU).</p>
<pre><code class="lang-auto">

import torch 
from torch import nn


device = torch.device('cuda') # crashes with cuda, works with cpu


class Model(nn.Module):

    def __init__(self):
        super().__init__()
        self.linear1 = nn.Linear(2, 16)
        self.linear2 = nn.Linear(2, 16)

    def forward(self, x, y):
        x = self.linear1(x)
        y = self.linear2(y)
        return torch.cat([x, y], dim=-1) # if we replace -1 with 1 works fine


model = Model().to(device)

data = [torch.randn(1, 2).to(device), torch.randn(1, 2).to(device)] 

traced = torch.jit.trace(model, data)

print(traced)
</code></pre>
<p>Surprisingly the above works with CPU backend but not with CUDA backend. It also works when <code>torch.cat(..., dim=1)</code> but crashes with a negative dimension refering to the same one <code>torch.cat(..., dim=-1)</code>.</p>
<p>Find the jit.trace error below (not very explanatory):</p>
<pre><code class="lang-auto">torch.jit.TracingCheckError: Tracing failed sanity checks!
Encountered an exception while running the trace with test inputs.
Exception:
        vector::_M_range_check: __n (which is 18446744073709551615) &gt;= this-&gt;size() (which is 2)
        The above operation failed in interpreter, with the following stack trace:

                               
</code></pre>

username: driazati
cooked: <p>This is a bug, you can track it in the corresponding <a href="https://github.com/pytorch/pytorch/issues/25648" rel="nofollow noopener">GitHub issue</a>, any updates / fixes that go in will get posted there.</p>
<p>As a workaround you can wrap the negative index around manually with something like <code>dim=len(x.shape) + (-1)</code></p>

username: imaluengo
cooked: <p>Yup sorry! Realised that later and filled an issue with it. <code>x.ndim - 1</code> should also work.</p>
Help on RNNs (specifically hidden layers on GRUs)

username: Vincent_Brisse
cooked: <p>Hello everyone,<br>
Let’s say I have the following rnn.<br>
self.rnn = torch.nn.GRU(<br>
input_size=1024,<br>
hidden_size=512,<br>
num_layers=2,<br>
batch_first=True<br>
)<br>
What does it mean ?<br>
I have a basic understanding of LSTMs. the input x_t goes through a GRU, then the hidden state h_t is updated to h_{t+1} and fed into the next unit with input x_{t+1]. In that regard, I understand the hidden size (the size of my input sequence) but I don’t understand the hidden_size and the number of layers. I’m sure just seeing a relevant picture would help me, can you help me out ?</p>

username: DoubtWang
cooked: <aside class="quote no-group" data-username="Vincent_Brisse" data-post="1" data-topic="55089">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vincent_brisse/48/14041_2.png" class="avatar"> Vincent_Brisse:</div>
<blockquote>
<p>sequence</p>
</blockquote>
</aside>
<p>the hidden size is the output size of rnn cell.<br>
if the num_layer &gt;= 2, the output of  the previous layer is the input of next layer.</p>

username: Vincent_Brisse
cooked: <p>Is this picture accurate ?<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/b/b67daed98787956336c596377a79b5a4a680e9d2.png" alt="image" data-base62-sha1="q2oaNHVw52PAeG3RnvG5UQNu4j8" width="688" height="312"></p>
<p>Is ‘g(1)’ always the same cell with the same weights. Its output is the hidden state ? Is the arrow on the right and the arrow on the top feeding the hidden state to both the next timestep and the next layer ?</p>

username: DoubtWang
cooked: <p>yes, the output of g^(1) is the hidden state.<br>
the arrow on the top feeds the hidden state to the next layer.<br>
For the arrow on the right, it feeds the hidden state and cell state (if cell is lstm cell); it feeds the hidden state (if cell is gru cell).<br>
u can read <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.LSTM" rel="nofollow noopener">this</a></p>

username: Vincent_Brisse
cooked: <p>Thank you I got it !</p>
How to train dataset (image with mask, image without mask )?

username: boyoung_Lim
cooked: <p>Dataset consists of an image with a ground truth mask and an image without a ground truth mask.</p>
<p>I want to use a mask rcnn network to learn masks for images with groud truth masks, and only classification and regression for images without ground truth masks.</p>
<p>Can I train using a mask rcnn network?</p>
<p>If I create a branch in the network, I don’t understand how to create it.</p>
<p>Thanks.</p>
Gradient descent back propagating for intermediate loss function

username: zack.zcy
cooked: <p>Hi there, recently, I’m writing a customized loss function, I’m not sure about whether propagating gradient through the whole network or just part of it.<br>
Specifically, I have an autoEncoder model, my loss function consists of 2 separate loss, where 1st part is taking the output of autoEncoder as input, whereas 2nd part is taking the intermediate output of the encoder network(without the decoder network) as its input. My question is that should I backprop gradient of 2nd loss through the decoder network?</p>
<pre><code>class autoEncoder(nn.Module):
    def __init__(self):
        super(autoEncoder,self).__init__()
    def forward(self,x):
        h = encoder(x)
        x = decoder(h)
        return x
</code></pre>
<p>For simplicity, encoder and decoder are 2 classes implementing the encoder network and decoder network respectively. with that defined, my loss function is formalized as follows:</p>
<pre><code>customized_loss  = loss_1(x) + loss_2(h) # loss_1 and loss_2 are torch.autograd.variable.Variable 
</code></pre>
<p>Should I write gradient descent as follows ?</p>
<pre><code>grad_params = torch.autograd.grad(loss_1,autoEncoder.parameters(),create_graph=True)
grad_params[-2].add_(torch.autograd.grad(loss_2,encoder.parameters(),create_graph=True))
torch.optim.SGD(grad_params).step()
</code></pre>
<p>or computes loss2 w.r.t he decoder network and update the whole network</p>
<pre><code>customized_loss.backward()
torch.optim.SGD().step()</code></pre>

username: zack.zcy
cooked: <p>Will pytorch automatically build a graph that traces the loss2 variable?</p>

username: smth
cooked: <p>you can simply do <code>customized_loss.backward()</code> and gradients wrt both loss1 and loss2 are propagated back to the parameters.</p>

username: zack.zcy
cooked: <p>thanks dude, wonderful framework!</p>

username: Shisho_Sama
cooked: <p>Did you get your answer?</p>
Beginner: Multiple Output's / Partial Dataset

username: brunoalano
cooked: <p>Hello,</p>
<p>I’m trying to build a model using PyTorch + pytorch_transformers to create it using BERT as pre-training step. The issue that I’ve different datasets, which all of them are on English, but they have partial intersected labels.</p>
<p>It’s possible to create a model that uses pre-trained BERT (or any other model), and feeds data from multiple datasets to predict multiple outputs?</p>
<p>Example, which I have 4 text datasets:<br>
Dataset A contains <code>[ ValueA, ValueB, ValueC ]</code><br>
Dataset B contains <code>[ ValueA, ValueB, ValueC, ValueD, ValueE, ValueF ]</code><br>
Dataset C contains <code>[ ValueA, ValueB ]</code><br>
Dataset D contains <code>[ ValueD, ValueE, ValueF ]</code></p>
<p>Since all of them are on English, I hope to use BERT to enchance the similarity between datasets.</p>
<p>Approaches that I thought:</p>
<ul>
<li>Create a general <code>y</code>, and add <code>0.</code> to empty fields which I don’t have for it. In this case, my prediction would be <code>[ ValueA, ValueB, ValueC, ValueD, ValueE, ValueF ]</code>
</li>
</ul>

username: ptrblck
cooked: <p>I’m not NLP expert, so take this with a grain of salt. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
<p>The use case sound generally like a multi-label classification, i.e. multiple classes can be active/inactive in the target.<br>
One approach would be to use a linear layer as your model output returning <code>[batch_size, nb_classes]</code> and pass it, along a target containing zeros and ones of the same shape, to <code>nn.BCEWithLogitsLoss</code>.</p>
<p>Let me know, if this would work or if I’m thinking too naively about your use case.</p>
VAE reconstruction loss

username: Rojin
cooked: <p>I have seen people writing the reconstruction loss in two different ways:</p>
<pre><code class="lang-auto">F.binary_cross_entropy(recon_x1, x1.view(-1, 784))

or 

F.binary_cross_entropy(recon_x1, x1.view(-1, 784), reduction = "sum")
</code></pre>
<p>I was wondering if there is a theoretical reason to use one over the other?</p>

username: spanev
cooked: <p>Hi <a class="mention" href="/u/rojin">@Rojin</a></p>
<p>I believe this comes from the fact that the <a href="https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence" rel="nofollow noopener">KL divergence</a> is an integral/sum.</p>
<p>So the sum reduction would be the more paper faithful approach, but having the PyTorch default (mean) will still work (only have downscaled gradients).</p>
Sequence of numbers

username: slavavs
cooked: <p>If I want to predict numbers (e.g. 12345-&gt; 6, 23456-&gt; 7, 34567-&gt; 8 …) and use the nn.transformer module. Do I need to do Ebidding or linear transform like Linear (1,256) or can I use d_model = 1, and just feed 1,2,3,4,5 … to the input?</p>
Problem in getting accuracy of different classes in image classification

username: mchowdhu
cooked: <p>I am doing an assignment related to  image classification.My dataset has 5 different classes(labels).I have to calculate accuracy of each class(labels) separately.But my problem is that whenever I tested my CNN model I obtained 100% accuracy for one label and accuracy of other labels are 0%. I could not make sense where is my implementation problem.<br>
Here is the code that i used for calculating accuracy of class labels</p>
<pre><code class="lang-auto"> classes = ['1', '2', '3', '4', '5']

    class_correct = list(0. for i in range(5))
    class_total = list(0. for i in range(5))

    net.eval()  # prep model for evaluation

    for batch in test_loader:
        data, target = batch['image'], batch['grade']

        if len(target.data) != BS:
            break
        # # forward pass: compute predicted outputs by passing inputs to the model
        output = net(data)
        # convert output probabilities to predicted class

        _, pred = torch.max(output, 1)
        # # compare predictions to true label
        correct = np.squeeze(pred.eq(target.data.view_as(pred)))
        # # calculate test accuracy for each object class
        for i in range(BS):
            label = target.data[i]
            class_correct[label] += correct[i].item()
            class_total[label] += 1

    for i in range(5):
        print('Test Accuracy of %5s: %2d%% (%2d/%2d)' % (
            classes[i], 100 * class_correct[i] / class_total[i],
            np.sum(class_correct[i]), np.sum(class_total[i])))

</code></pre>
<p>This is my results that I obtained</p>
<pre><code class="lang-auto">Accuracy of the network (overall): 46 %
    Test Accuracy of     1: 100% (652/652)
    Test Accuracy of     2:  0% ( 0/279)
    Test Accuracy of     3:  0% ( 0/459)
    Test Accuracy of     4:  0% ( 0/263)
    Test Accuracy of     5:  0% ( 0/47)
</code></pre>

username: ptrblck
cooked: <p>The calculation looks alright.<br>
Your model might just overfit to class0, thus predicting every sample as this class.<br>
Check the prediction distribution using</p>
<pre><code class="lang-python">print(pred.unique(return_counts=True))
</code></pre>
<p>to see, whether your model predicts other classes.</p>

username: mchowdhu
cooked: <p>Whenever I called</p>
<pre><code class="lang-auto">print(pred.unique(return_counts=True))
</code></pre>
<p>I got the below results</p>
<pre><code class="lang-auto">(tensor([0]), tensor([50]))</code></pre>

username: ptrblck
cooked: <p>Even though your class distribution is not very imbalanced, it seems your model is still overfitting to this class.<br>
You could e.g. add the <code>weight</code> parameter to your criterion and try to counter this effect.</p>
Help needed: VGG16 with tensor decomposition got nan parameters and loss

username: attomos
cooked: <p>I’ve been experimenting with tensor decomposition in VGG16 model based on this <a href="https://github.com/jacobgil/pytorch-tensor-decompositions" rel="nofollow noopener">repo</a>.</p>
<p>What I’m trying to do is split conv layer into smaller parts, apply tucker decomposition in each of them, and put them back to conv layer as in baseline model.<br>
In the baseline model, it takes each Conv2d layer from VGG model and break them into 3 smaller Conv2d layers.</p>
<p>I printed model parameters before and after training. It was fine before training.<br>
After the training, loss and parameters became nan.</p>
<p>You can check out my code in this <a href="https://colab.research.google.com/drive/1PpE9seTtHM6Ejkoni2QIzSfnUv_-Is2K" rel="nofollow noopener">colab</a>.</p>
<p>Any suggestions would be much appreciated.</p>
<p>Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Torch 1.1.0.post2 doesn't contain "manylinux" wheel files. Who can push them?

username: lapolonio
cooked: <p>I’m using snorkel on a centos machine. And it’s trying to download torch 1.1.0.post2 but it’s not installing properly.</p>
<p>According to <a href="https://pypi.org/project/torch/1.1.0.post2/#files" rel="nofollow noopener">https://pypi.org/project/torch/1.1.0.post2/#files</a><br>
The manylinux versions don’t exist for the different python versions. Who can push them?</p>
<p>Compare:<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://pypi.org/static/images/favicon.6a76275d.ico" class="site-icon" width="16" height="16">
      <a href="https://pypi.org/project/torch/1.1.0/#files" target="_blank" rel="nofollow noopener">PyPI</a>
  </header>
  <article class="onebox-body">
    <img src="https://pypi.org/static/images/twitter.c0030826.jpg" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://pypi.org/project/torch/1.1.0/#files" target="_blank" rel="nofollow noopener">torch</a></h3>

<p>Tensors and Dynamic neural networks in Python with strong GPU acceleration</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://pypi.org/static/images/favicon.6a76275d.ico" class="site-icon" width="16" height="16">
      <a href="https://pypi.org/project/torch/1.1.0.post2/#files" target="_blank" rel="nofollow noopener">PyPI</a>
  </header>
  <article class="onebox-body">
    <img src="https://pypi.org/static/images/twitter.c0030826.jpg" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://pypi.org/project/torch/1.1.0.post2/#files" target="_blank" rel="nofollow noopener">torch</a></h3>

<p>Tensors and Dynamic neural networks in Python with strong GPU acceleration</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>