akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
acGAN - CUDA out of memory

username: pitty
cooked: <p>Hello!</p>
<p>I am trying to run the following code: <a href="https://github.com/Sirius79/acGAN/blob/master/train.py" rel="nofollow noopener">https://github.com/Sirius79/acGAN/blob/master/train.py</a> for more epochs and I get CUDA out of memory error after around 60.</p>
<p>I am a really noob in Torch and I did not write that code. After researching a bit I think some tensors are not allocated the way they should be so the memory usage increases with each iteration, but I cannot find the problem myself (tried, but it takes 3 hours to get to the error so changing every line of code in every possible way does not work).</p>
<p>Any help would be greatly appreciated.</p>
<p>Thanks</p>
Libtorch for Arm Core is not yet?

username: bemorept
cooked: <p>Hi, All.</p>
<p>Libtorch for Arm Core is not yet?<br>
In Raspberry Pi 3B, I just failed install.</p>
<p>I wonder …</p>
<p>Thanks in advance.</p>
<p><span class="mention">@bemoregt</span>.</p>
What is the best way to use golden feature in machine learning model?

username: aerinykim
cooked: <p>Let’s say I have a classification model. And my job is to predict the correct class out of 30 different classes. The current accuracy is 70%.</p>
<p>The thing is: I <em>have to</em> consume another team’s classification result which is 80% accurate. So I’m using their prediction result as a feature. I’ll call it “golden feature”. Let’s say I’m aiming &gt;80% accuracy with the golden feature.</p>
<p>Here is my current approach:</p>
<p>(I’m using Deep Learning.) I have several features and each feature has its own weight. I also create a weight vector for one hot vector (1 by 30) of “golden feature” and train all weights together. However the result doesn’t seem to provide much.</p>
<p>I thought about the reason why and realized that the learned vector (30 by n) won’t be that meaningful. They would be just positive numbers.<br>
(Please yell at me if my reasoning is wrong!)</p>
<p>Has anyone faced the similar problem? Any suggestion will be highly appreciated!<br>
The method that you suggest doesn’t have to be Deep Learning approach.</p>

username: ptrblck
cooked: <p>You could try to use the output of the penultimate layers, concatenate them, and train a final classifier. This would be similar to the vanilla fine tuning approach, but with multiple inputs.</p>
<p>If this doesn’t provide much benefit, I would go for e.g. XGBoost and try to create a model ensemble out of your trained models.</p>
Replace input channel and kernel size in pre-trained model

username: vivekstorm
cooked: <p>I tried using pre-trained resnet-50 with image size of 400X400 and modified Conv1 input channel and kernel size. I didn’t get any error. How will be the weights initialized in this case.</p>
<pre><code class="lang-auto">import torch
import torchvision
import torch.nn as nn
from torchsummary import summary
from torchvision import datasets, models, transforms

model_ft = models.resnet50(pretrained=False)
model_ft.conv1=nn.Conv2d(1, 64, kernel_size=(3, 3), stride=(2, 2), padding=(3, 3), bias=False)

</code></pre>
<p>When I tried to print the summary I got</p>
<pre><code class="lang-auto">print(summary(model_ft.cuda(),input_size=(1,400,400)))
------------------------------------------------------------------------
        Layer (type)               Output Shape         Param #
==========================================
            Conv2d-1         [-1, 64, 202, 202]             576
       BatchNorm2d-2         [-1, 64, 202, 202]             128
              ReLU-3         [-1, 64, 202, 202]               0
         MaxPool2d-4         [-1, 64, 101, 101]               0
            Conv2d-5         [-1, 64, 101, 101]           4,096

</code></pre>
<p>Without modifying anything in pre-trained model, I got summary as</p>
<pre><code class="lang-auto">model_ft = models.resnet50(pretrained=True)
print(summary(model_ft.cuda(),input_size=(3,400,400)))
----------------------------------------------------------------
        Layer (type)               Output Shape         Param #
=========================================
            Conv2d-1         [-1, 64, 200, 200]           9,408
       BatchNorm2d-2         [-1, 64, 200, 200]             128
              ReLU-3         [-1, 64, 200, 200]               0
         MaxPool2d-4         [-1, 64, 100, 100]               0
            Conv2d-5         [-1, 64, 100, 100]           4,096

</code></pre>
<p>If I try to remove/change layers in pre-trained model, Pytorch doesn’t throw any error. How will be the weights are initilized after changing the pre-trained model?</p>
Replace some layer in pytorch

username: dingyongchao
cooked: <p>I define some layer by myself, and I want to replace some layers in origin Net using my layer.<br>
For Example:<br>
Net define:</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.conv2 = nn.Conv2d(6, 16, 3)
        self.conv3 = nn.Conv2d(16, 32, 3)
        self.pool = nn.MaxPool2d(2, 2)
        self.fc1 = nn.Linear(32, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = self.pool(F.relu(self.conv3(x)))
        x = x.view(-1, 32)
        x = self.fc1(x)
        return x
</code></pre>
<p>In this Net, F.relu is used. However, I want to replace the relu in this Net.<br>
Therefore, if I print this Net() :</p>
<pre><code class="lang-auto">Net(
  (conv1): Conv2d(3, 6, kernel_size=(5, 5), stride=(1, 1))
  (conv2): Conv2d(6, 16, kernel_size=(3, 3), stride=(1, 1))
  (conv3): Conv2d(16, 32, kernel_size=(3, 3), stride=(1, 1))
  (pool): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  (fc1): Linear(in_features=32, out_features=10, bias=True)
)
</code></pre>
<p>which I found that Relu operation not existed.</p>
<p>So I want to know If I want to modify relu operation in Net(Net can not be rewrited), How can I deal with it?<br>
if jit.script can help me to do this process?<code>Preformatted text</code></p>

username: ptrblck
cooked: <p>Since relu is used as a functional call, the cleanest way would be to override the class and reimplement the <code>forward</code> method with your new non-linearity.<br>
By doing so, you could also create the new activation function as an <code>nn.Module</code>, so that it can be replaced easily, if that’s your use case.</p>

username: dingyongchao
cooked: <p>The situation or the request for me is that I get an model and Net define from others, And I need to design  a framework to replace some operation such as relu or others. So modify net define by hand is not realistic for me.<br>
If there has any other advance?</p>

username: ptrblck
cooked: <p>You could try to override the <code>F.relu</code> directly with your desired method, but I would consider this quite a hack, since this will invisibly redefine this method.</p>
Torchvision 0.3 NMS

username: jianfei
cooked: <p>Hello, I’m not clear how to use the NMS from torchvision 0.3 to postprocess images in batch, is there any demo to share?</p>

username: tom
cooked: <p>torchvision itself ships detection models using this via the <a href="https://github.com/pytorch/vision/blob/master/torchvision/ops/boxes.py#L30" rel="nofollow noopener">batched NMS</a> utility function.</p>
<p>Best regards</p>
<p>Thomas</p>
Transformations when Dataset returns multiple images

username: lorenzo_fabbri
cooked: <p>My custom Dataset returns 3 objects: 2 images (as a dictionary) and the label (the same for both images):</p>
<pre><code class="lang-auto">def __getitem__(self, idx):
    ...
    return {'x1': img1, 'x2': img2}, label
</code></pre>
<p>I’m wondering whether the transformations I have defined are applied to both images:</p>
<pre><code class="lang-auto">transform_train = transforms.Compose([
    transforms.RandomRotation(degrees=30), 
    transforms.RandomVerticalFlip(), 
    transforms.RandomHorizontalFlip(), 
    transforms.ToTensor(), 
    transforms.Normalize(means, stds)
])
</code></pre>
<p>Is this the case or do I have to modify them?</p>

username: hktxt
cooked: <p>The class of transforms should be a arguments passed to your custom Dataset.<br>
Then in the func of <strong>getitem</strong>(), you should apply transforms to each image.<br>
see tutorials here: <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html#dataset-class" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/data_loading_tutorial.html#dataset-class</a></p>

username: lorenzo_fabbri
cooked: <p>Thank you, I understand. I forgot to apply the transformations in my custom Dataset class…<br>
But now I have another problem. My images have more than 4 channels, and when I apply <code>ToPILImage</code>, the output has only 3 channels. Is it possible to have a PIL image with the original number of channels?</p>
<p>For more details.<br>
Here’s how I create a tensor with 6 channels in my custom Dataset class:</p>
<pre><code class="lang-auto"># Iterate over the files corresponding to each channel of 1 image
for path in paths:
    path = self.path_imgs+path

    with pil.Image.open(path) as img:
        # Append channel
        channels.append(tv.transforms.ToTensor()(img))

if self.transform:
    to_append = self.transform(t.cat(channels, dim = 0))
</code></pre>
<p>Then my transformation is:</p>
<pre><code class="lang-auto">transform_train = transforms.Compose([
    transforms.ToPILImage(), 
    transforms.RandomRotation(degrees=30), 
    transforms.RandomVerticalFlip(), 
    transforms.RandomHorizontalFlip(), 
    transforms.ToTensor(), 
    transforms.Normalize(means, stds)
])
</code></pre>
Multi threaded ops on CPU

username: soldierofhell
cooked: <p>Hi, is <a href="https://github.com/pytorch/pytorch/blob/master/c10/core/thread_pool.h" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/c10/core/thread_pool.h</a> an equivalent of tensorflow threadpool? Why it’s not used e.g. in torchvision cpu ops? Is there any doc how to use it?<br>
Regards,</p>

username: tom
cooked: <p>I think <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/Parallel.h" rel="nofollow noopener">ATen/Parallel.h</a> is a reasonable interface for intra-op parallelism.</p>
<p>Regarding the ops: The extension ops implemented in torchvision are for FastRCNN/MaskRCNN: NMS, ROIAlign, ROIPool.</p>
<p>I do not have any special insight, but my impression from timing MaskRCNN-Benchmark with in the FBNet variant is that the ops implemented in torchvision itself represent a relatively small part of the overall calculation of a FastRCNN model, with the bulk of the time being spent in convolutions for the backbone. In addition, the CPU implementation seems to not be particularly optimized but mainly provided for completeness (and probably isn’t an urgent optimization target). If you were serious about optimizing it, you would likely want to look at vectorization (AVX2) before threading.</p>
<p>Best regards</p>
<p>Thomas</p>

username: soldierofhell
cooked: <p>Thank you <a class="mention" href="/u/tom">@tom</a> for the relevant header.<br>
I mentioned torchvision, because I’m looking for reference example how to parallelize for loops in custom ops, and different variants of ROI transformations are typical examples. Of course there is some heavy stuff in the base model but I assume it’s tuned to the maximum. Just want to speed up post processing custom/opencv routines.</p>
<p>Regards,</p>
Where did GRU implement?

username: HsuLinNwe
cooked: <p>I want to know how GRU is implemented in pytorch. But I can’t find the implementation of GRU although I looked in <a href="https://github.com/pytorch/pytorch/tree/949559552004db317bc5ca53d67f2c62a54383f5/aten/src/THNN" rel="nofollow noopener">here</a>. Can anyone tell me where did GRU implement in pytorch?</p>

username: Prerna_Dhareshwar
cooked: <p>Hi,</p>
<p>Are you looking for the source code? If so you can find it <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/rnn.py" rel="nofollow noopener">here</a>.</p>

username: tom
cooked: <p>The “native” implantations are in<br>
<a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/RNN.cpp" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/RNN.cpp</a> and<br>
<a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/cuda/RNN.cu" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/cuda/RNN.cu</a><br>
There also are CuDNN bindings…</p>

username: HsuLinNwe
cooked: <p>Thank <a class="mention" href="/u/tom">@tom</a> for your reply.<br>
I am a novice in C+ and although I looked for the how does narrow works, I still do not know clearly what does hidden_slice return.</p>
<pre><code class="lang-auto">Tensor hidden_slice(const Tensor&amp; t, int64_t start, int64_t end) {
  return t.narrow(0, start, end - start);
}
tpair_of&lt;Tensor&gt; hidden_slice(const tpair_of&lt;Tensor&gt;&amp; t, int64_t start, int64_t end) {
  return std::make_tuple(hidden_slice(std::get&lt;0&gt;(t), start, end),
                         hidden_slice(std::get&lt;1&gt;(t), start, end));
}
</code></pre>
<p>Can you please explain me what does hidden_slice return.</p>

username: tom
cooked: <p>So it’s overloaded: calling it with a pair (second variant) will return a tuple with sliced elements by calling the first variant on each of the two members.</p>
<p>Best regards</p>
<p>Thomas</p>

username: HsuLinNwe
cooked: <p>Thank you <a class="mention" href="/u/tom">@tom</a> <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"><img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
Does the Transfer learning or fine-tuned Pytorch model could not converted to Caffe model or ONNX?

username: bemorept
cooked: <p>Hi, All.</p>
<p>Does the Transfer learning or fine-tuned Pytorch model could not converted to Caffe model or ONNX?</p>
<p>Or does the conversion work well, and does not model the inference code optimization?</p>
<p>Have you ever tried Caffe, or onnx Inference, as a transfer learning model?</p>
Dot: Expected 1-D argument tensor, but got 2-D

username: an99
cooked: <p>Hi  their !!<br>
I have some problem in my code  it torch.dot product</p>
<pre><code class="lang-auto">ypo=torch.tensor([[4,6,7],[4,6,7],[4,6,7]],dtype=torch.float)
yo=torch.tensor([4,6,7],dtype=torch.float)
xyz=torch.dot(yo,ypo)

</code></pre>
<p>this give  error   it is<br>
dot: Expected 1-D argument tensor, but got 2-D<br>
plz help me …<img src="https://discuss.pytorch.org/images/emoji/apple/smiling_face_with_three_hearts.png?v=9" title=":smiling_face_with_three_hearts:" class="emoji" alt=":smiling_face_with_three_hearts:"></p>

username: tom
cooked: <p>Are you lookiing for torch.matmul?</p>
<p>Best regards</p>
<p>Thomas</p>

username: Pranavan_Theivendira
cooked: <aside class="quote no-group" data-username="an99" data-post="1" data-topic="51208">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/f14d63/48.png" class="avatar"> an99:</div>
<blockquote>
<p>ypo=torch.tensor([[4,6,7],[4,6,7],[4,6,7]],dtype=torch.float) yo=torch.tensor([4,6,7],dtype=torch.float) xyz=torch.dot(yo,ypo)</p>
</blockquote>
</aside>
<p>Hi <a class="mention" href="/u/tom">@tom</a>,</p>
<p>Just curious, there is another function <code>torch.mm</code> which can do the same multiplication. Which one is the updated one?</p>
<p>Thanks</p>

username: tom
cooked: <p>For matrix x vector <code>torch.mv</code> would work, too.</p>
<p>Best regards</p>
<p>Thomas</p>
The difference between actor-critic example and A2C?

username: zuoxingdong
cooked: <p>In PyTorch example repo there is an example code of <a href="https://github.com/pytorch/examples/blob/master/reinforcement_learning/actor_critic.py" rel="nofollow noopener">actor-critic</a>. I am quite curious that what is the difference between this code and <a href="https://github.com/openai/baselines/tree/master/baselines/a2c" rel="nofollow noopener">A2C</a> ? Because it also learns a baseline for advantage estimate which sounds like what A2C is doing ?</p>

username: keerthanpg
cooked: <pre><code class="lang-auto">for (log_prob, value), r in zip(saved_actions, rewards):
        reward = r - value.data[0]
        policy_losses.append(-log_prob * reward)
        value_losses.append(F.smooth_l1_loss(value, Variable(torch.Tensor([r]))))
</code></pre>
<p>So in the current code, if you see how the loss is calculated, you’d see that the critic is taught to learn the reward of a particular state, action pair, given by r, or the action-value function rather than the advantage function. In A2C, the critic would learn the advantage function.</p>

username: huiqing
cooked: <p>actor-critic is a kind of method.Advantage actor-critic(A2C) is a good implement of this method.<br>
I don’t think what you have said is all right. Actually the <strong>reward</strong> in the code you stick is the advantage function.In the latest code, the <strong>reward</strong> has been changed to <strong>advantage</strong>.And the advantage function can be represented as <img src="https://discuss.pytorch.org/uploads/default/original/2X/b/b0c50e8079cf25dfe13466cf2f029308c9ec618c.png" alt="QGRVDKGOS~%7D896%24DL6J0YKP" data-base62-sha1="pdMe3vITdXkXpqGQ2iYZws2bqi0" width="158" height="25">，which is equal to the code : <strong>advantage = r - value.data[0]</strong></p>
CNN RNN Hybrid - sequences of variable length

username: Felix_Kreuk
cooked: <p>Hi all,</p>
<p>I am currently trying to train a network for a regression task, basically I need to map a variable-length sequence (say a spectrogram) to a fixed size vector. My baseline model was a <code>GRU</code> (many-to-one) which mapped the sequence to a vector, and then fed it to a <code>Linear</code> layer for the output, this didn’t go very well. Which led me to believe that perhaps I should shrink the spectrogram before handing it over to the <code>GRU</code>. This actually helped a lot.</p>
<p>My problem is, with the baseline <code>GRU</code> model I just used <code>PackedSequence</code> because I knew the original lengths of the sequences (which are need to create a <code>PackedSequence</code>). But now, after the sequences first pass through a <code>conv2d</code> layer, I “lose” that information and no longer know the length of sequences. So my question is - what it the common practice in such cases? I am open to other architectures as well btw.</p>
<p>Thanks,<br>
Felix.</p>

username: Enumaris
cooked: <p>Maybe I’m misunderstanding something, but why would passing a sequence through a convolutional layer make you “lose” information about the length of that sequence? With a given series of convolutions, you should be able to quite easily figure out the output shape given the input shape?</p>

username: Felix_Kreuk
cooked: <p>So let’s say I have sequences of different length, I need to zero-pad them all to the same length to work in batches. So for each sequence I “remember” its length. Let’s say for example I have a sequence tensor of 300 x 100, and I “remember” that its actual length is 230, meaning there are 70 zero padded time frames. Now, after I pass that through conv+pool this shrinks to a new size, the old length of 230 is not longer relevant. Did I manage to explain myself? I feel like maybe this is a bit tricky, or I’m missing something.</p>

username: job28
cooked: <p>Hey <a class="mention" href="/u/felix_kreuk">@Felix_Kreuk</a>,<br>
Did you manage to find a solution to this, I am in the exact same position now.</p>
CUDA error: an illegal memory access with C++ API while calling .to(k::CUDA) from cuda code

username: makezur
cooked: <p>Hi!</p>
<p>I have a class with some at::Tensor fields and some of its methods are implemented in cuda.<br>
At some point an at::Tensor field has to be moved to cuda in order to execute a cuda kernel. During such an operation:</p>
<pre><code class="lang-auto">tensor = tensor.to(at::kCUDA);
</code></pre>
<p>The following error arises:</p>
<pre><code class="lang-auto">CUDA error: an illegal memory access was encountered (copy_from_cpu at /opt/conda/conda-bld/pytorch_1549628766161/work/aten/src/ATen/native/cuda/Copy.cu:178)
frame #0: c10::Error::Error(c10::SourceLocation, std::string const&amp;) + 0x45 (0x7f565fc4ccf5 in ../lib/python3.6/site-packages/torch/lib/libc10.so)
frame #1: (anonymous namespace)::copy_from_cpu(at::Tensor&amp;, at::Tensor const&amp;) + 0x4ab (0x7f566510b86b in ../lib/python3.6/site-packages/torch/lib/libcaffe2_gpu.so)
frame #2: void (anonymous namespace)::_copy__cuda&lt;int&gt;(at::Tensor&amp;, at::Tensor const&amp;, bool) + 0xd35 (0x7f56651a7245 in ../lib/python3.6/site-packages/torch/lib/libcaffe2_gpu.so)
frame #3: at::native::_s_copy__cuda(at::Tensor&amp;, at::Tensor const&amp;, bool) + 0x1e5 (0x7f566510bbe5 in ../lib/python3.6/site-packages/torch/lib/libcaffe2_gpu.so)
frame #4: at::CUDAIntType::s_copy_(at::Tensor&amp;, at::Tensor const&amp;, bool) const + 0x62 (0x7f5663ecd9a2 in ../lib/python3.6/site-packages/torch/lib/libcaffe2_gpu.so)
frame #5: at::TypeDefault::copy_(at::Tensor&amp;, at::Tensor const&amp;, bool) const + 0x108 (0x7f56607896c8 in ../lib/python3.6/site-packages/torch/lib/libcaffe2.so)
frame #6: at::TypeDefault::copy(at::Tensor const&amp;, bool, c10::optional&lt;c10::Device&gt;) const + 0x322 (0x7f5660751732 in ../lib/python3.6/site-packages/torch/lib/libcaffe2.so)
frame #7: &lt;unknown function&gt; + 0x719737 (0x7f5660577737 in ../lib/python3.6/site-packages/torch/lib/libcaffe2.so)
frame #8: at::native::to(at::Tensor const&amp;, c10::Device, c10::ScalarType, bool, bool) + 0x4c8 (0x7f566057a148 in ../lib/python3.6/site-packages/torch/lib/libcaffe2.so)
frame #9: at::TypeDefault::to(at::Tensor const&amp;, c10::Device, c10::ScalarType, bool, bool) const + 0x1b (0x7f5660717d0b in ../lib/python3.6/site-packages/torch/lib/libcaffe2.so)
</code></pre>
<p>Am I not supposed to call .to() in .cu files?</p>
<p>I use the latest stable PyTorch 1.0.1 and CUDA 9.0</p>

username: yf225
cooked: <p>Can you try moving the tensor to CUDA before calling the CUDA kernel?</p>
Potential bug in ImageFolder?

username: numpee
cooked: <p>I’m using the ImageFolder class provided by Pytorch to load a dataset. The dataset is under a the folder</p>
<pre><code class="lang-auto">/home/user/faces_emore/imgs/
</code></pre>
<p>which contains all the subfolders to the images.</p>
<p>While training, my program exited with an unusual error:</p>
<pre><code class="lang-auto">FileNotFoundError: [Errno 2] No such file or directory: '/home/user/data/faces_emore/imgs/1:85/85337.jpg'
</code></pre>
<p>Which is completely out of the blue because this occurred in the second epoch - after the first epoch was completely with no errors. I searched for ‘85337.jpg’ in my dataset and found that it is under the subfolder</p>
<pre><code class="lang-auto">'/home/user/data/faces_emore/imgs/1285/85337.jpg'
</code></pre>
<p>meaning that the ‘2’ was somehow replaced with a ‘:’ during training…?<br>
I’m curious if anyone has encountered this or anything similar.</p>
<p>Thanks</p>

username: InnovArul
cooked: <p>Are you able to reproduce it or does it occur sporadically?</p>

username: numpee
cooked: <p>Hi, I just found this bug, so I haven’t been able to reproduce it. I’ll try training without changing anything and see what happens. Only problem is that it takes around 6 hours to train a single epoch, so it’ll probably take some time.</p>

username: InnovArul
cooked: <p>If it’s a bug with ImageFolder,  you do not need to run the training (or access the model).  You could just run the ‘for’ loop over the dataset and spot the error.</p>

username: numpee
cooked: <p>I’ve run a couple more runs and haven’t been able to reproduce it. I guess it must have been a random system error.</p>
Question about AdaptiveMaxPooling implementation

username: crcrpar
cooked: <p>Is it OK to accept nan value as max value?<br>
I do not think this natural, so could anyone tell me why?</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/8b798f43e30d17f34b0af11c3fd17f7832003c46/aten/src/ATen/native/AdaptiveMaxPooling2d.cpp#L70" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/8b798f43e30d17f34b0af11c3fd17f7832003c46/aten/src/ATen/native/AdaptiveMaxPooling2d.cpp#L70" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/8b798f43e30d17f34b0af11c3fd17f7832003c46/aten/src/ATen/native/AdaptiveMaxPooling2d.cpp#L70</a></h4>
<pre class="onebox"><code class="lang-cpp"><ol class="start lines" start="60" style="counter-reset: li-counter 59 ;">
<li>
</li>
<li>/* compute local max: */</li>
<li>int64_t maxindex = -1;</li>
<li>scalar_t maxval = -std::numeric_limits&lt;float&gt;::max();</li>
<li>int ih, iw;</li>
<li>for(ih = 0; ih &lt; kH; ih++)</li>
<li>{</li>
<li>  for(iw = 0; iw &lt; kW; iw++)</li>
<li>  {</li>
<li>    scalar_t val = *(ip + ih*istrideH + iw*istrideW);</li>
<li class="selected">    if ((val &gt; maxval) || std::isnan(val))</li>
<li>    {</li>
<li>      maxval = val;</li>
<li>      maxindex = (ih+istartH)*isizeW + (iw+istartW);</li>
<li>    }</li>
<li>  }</li>
<li>}</li>
<li>
</li>
<li>/* set output to local max */</li>
<li>*op = maxval;</li>
<li>
</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


username: yf225
cooked: <p>As this is not a libtorch-specific question, would you like to post it in <a href="https://github.com/pytorch/pytorch/issues" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues</a> to reach a wider audience / get more help on the question?</p>
R-CNN better understanding

username: Shubham_Sharma
cooked: <p>what is xavier_uniform_, I do understand it is similar to Gaussian distribution but what is the difference between xavier_uniform_ and gaussian distribution<br>
from below code it picked it up</p>
<p>def init_conv2d(self):<br>
“”"<br>
Initialize convolution parameters.<br>
“”"<br>
for c in self.children():<br>
if isinstance(c, nn.Conv2d):<br>
nn.init.xavier_uniform_(c.weight)<br>
nn.init.constant_(c.bias, 0.)</p>

username: Pranavan_Theivendira
cooked: <aside class="quote no-group" data-username="Shubham_Sharma" data-post="1" data-topic="51215">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shubham_sharma/48/13877_2.png" class="avatar"> Shubham_Sharma:</div>
<blockquote>
<p>xavier_uniform_</p>
</blockquote>
</aside>
<p>Hi,</p>
<p>Xavier initialization is a way to initialize the weights with the right values.(They are not big as well as not small.) You can find a detailed explanation from andy’s blog.</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://andyljones.tumblr.com/post/110998971763/an-explanation-of-xavier-initialization">
  <header class="source">
      <img src="https://64.media.tumblr.com/avatar_d867a4dd8ef1_128.pnj" class="site-icon" width="128" height="128">

      <a href="https://andyljones.tumblr.com/post/110998971763/an-explanation-of-xavier-initialization" target="_blank" rel="noopener nofollow ugc">Tumblr</a>
  </header>

  <article class="onebox-body">
    

<h3><a href="https://andyljones.tumblr.com/post/110998971763/an-explanation-of-xavier-initialization" target="_blank" rel="noopener nofollow ugc">An Explanation of Xavier Initialization</a></h3>

  <p>If you're having trouble viewing the formulas below, try turning off Adblock (thanks /u/BeatLeJuce)
If you work through the Caffe MNIST tutorial, you'll come across this curious line
weight_filler {...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Thanks</p>
Can one change the learning rate of a layer after the optimizer is initialised?

username: divinho
cooked: <p>I can’t find an example anywhere. Additionally I can’t find out what exactly <code>optimizer.param_groups</code> returns, so I don’t know whether I can filter by name to change the learning rate.</p>

username: tom
cooked: <p>The param groups are dictionaries with the state information. If you only have one param group (this is the case if you just initialized the optimizer with <code>model.parameters()</code>) then <code>optimizer.param_groups[0]['lr']</code> is the learning rate.<br>
Parameter groups are useful e.g. when you want different learning rates for different parameters. For example fast.ai advocates that for finetuning.</p>
<p>Best regards</p>
<p>Thomas</p>

username: divinho
cooked: <p>Okay I see so I must split things up when initializing. Thank you.</p>
Stacking frames in BCHW format for DQN

username: Milky
cooked: <p>Hello so I have to use BCHW format for my dqn and I have it working with 1 frame, and it’s a “grayscale” image, so no rgb values. I just have:<br>
state = getState()  -&gt; 50x50 matrix<br>
state= [None, None, :, : ]</p>
<p>However, I want to try stacking the frames and I’m not sure how I would go about doing that. So would I need a new array to store 4 bchw format “states” or could I give the number of layers as the channel in one bchw format state?</p>
<p>Also, would stacking affect anything else in the code that I should be aware of, like in the network?</p>
Pip install does not work for jupyter notebook

username: Yifan_Xu
cooked: <p>I initially tried <code>pip3 install torch</code> but when I import torch in jupyter notebook it said that no module found. I then used conda to install it and import worked.  Does anyone knows why would pip fail for any specific reasons?</p>

username: tom
cooked: <p>There are detailed pip commands on the <a href="https://pytorch.org/get-started/locally/" rel="nofollow noopener">Get started</a> page, maybe the download.pytorch.org-hosted versions for CUDA 10 or CPU work better. I always install PyTorch via <code>pip3</code>.</p>
<p>Best regards</p>
<p>Thomas</p>
nn.LSTMCell inside nn.LSTM?

username: gt_tugsuu
cooked: <p>Does this <code>rnn = nn.LSTM(input_size, hidden_size)</code> mean, it has  <code>hidden_size</code> number of <code>nn.LSTMCell()</code> cells inside?</p>

username: jlquinn
cooked: <p>No, hidden_size is the hidden dimension of the LSTM cell.</p>
<p>LSTM is functionally equivalent to writing a loop over your input and feeding it to LSTMCell (modulo the API differences).  LSTM exists because the cuDNN library provides a more efficient implementation when you know the full input at the start of the computation.  Also, LSTM is more efficient when you are using more than one layer of LSTM cells.</p>

username: gt_tugsuu
cooked: <p><a class="mention" href="/u/jlquinn">@jlquinn</a><br>
Thanks for replying.<br>
So does it mean it’s just one LSTMCell? (in this case)</p>
Change Learning rate during training with custom values

username: wally88
cooked: <p>Hi all,</p>
<p>I am wondering if there is a way to set the learning rate each epoch to a custom value.</p>
<p>for instance in Matconvent you can specify learning rate as  <code>LR_SCHEDULE = np.logspace(-3, -5, 120)</code>  to have it change from .001 to .00001 over 120 training epochs, for instance.</p>
<p>is there something similar I can do in Pytorch?</p>
<p>my first idea is to define the following function and then re-define the optimizer each epoch</p>
<pre><code>def scheduler(optimizer,lr):
    for param_group in optimizer.param_groups:
        param_group['lr'] = lr
    return optimizer
</code></pre>
<p>so then</p>
<pre><code>for epoch in range(EPOCHS):
        lr = LR_SCHEDULE[epoch]
        optimizer = scheduler(optimizer,lr)
</code></pre>
<p>could this work?</p>
<p>thanks</p>

username: ptrblck
cooked: <p><code>torch.optim.lr_scheduler</code> is basically doing the same update step as your <code>scheduler</code> code (besides some other checks).<br>
Have a look at the implemented <a href="https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate" rel="nofollow noopener">lr_schedulers</a> to avoid rewriting them.</p>

username: wally88
cooked: <p>Thank you, I am aware of the <code>torch.optim.lr_scheduler</code>, I was more looking for something I can customize if needed instead of using the implemented versions</p>

username: Yinda_XU
cooked: <p><em>torch.optim.lr_scheduler</em> seems to adjust LR in a “relative” fashion, seeing that its method <em>get_lr()</em> do not take any argument. However, in most of my cases, I wish to do in an “absolute” fashion that set LR given current epoch and current iteration (yes I adjust the LR each iteration). In this case, such a wheel mentioned above is actually doing great for me.</p>
Different results on CPU and GPU of RuntimeError: hidden of RNN is not contiguous

username: ShawnGuo
cooked: <p>For me, this is a quite interesting problem. I ran the same codes but got different results on CPU and GPU. To be specific, the following codes can run on CPU without any error raised. However, when running on GPU, the following error is raised:</p>
<pre><code class="lang-auto">RuntimeError: rnn: hx is not contiguous
</code></pre>
<p>The codes of my class are given as follows:</p>
<pre><code class="lang-auto">class EncoderLSTM(nn.Module):
    def __init__(self, voc_size, hidden_size=HIDDEN_SIZE, max_length=MAX_LENGTH+2):
        super(EncoderLSTM, self).__init__()
        self.hidden_size = hidden_size
        self.max_length = max_length

        self.memorising = nn.Embedding(voc_size, self.hidden_size)
        self.attn = Attn(hidden_size)
        self.dropout = nn.Dropout(DROPOUT_RATIO)
        self.lstm = nn.LSTM(hidden_size, hidden_size)
        
        self.init_hidden = self.init_hidden_and_cell()
        self.init_cell = self.init_hidden_and_cell()

    def forward(self, embedded_input_var):
        batch_size = embedded_input_var.shape[0]

        # Initialise the initial hidden and cell states for encoder
        last_hidden = self.init_hidden.expand(-1, batch_size, -1)
        last_cell = self.init_cell.expand(-1, batch_size, -1)
        
        # Forward pass through LSTM
        for t in range(NUM_WORD):
            # Calculate attention weights from the current LSTM input
            attn_weights = self.attn(last_hidden, embedded_input_var)
            # Calculate the attention weighted representation
            r = attn_weights.bmm(embedded_input_var).transpose(0, 1)
            # Forward through unidirectional LSTM
            lstm_output, (lstm_hidden, lstm_cell) = self.lstm(r, (last_hidden, last_cell))

        # Return hidden and cell state of LSTM
        return lstm_hidden, lstm_cell

    def init_hidden_and_cell(self):
        return nn.Parameter(torch.zeros(1, 1, self.hidden_size, device=DEVICE))
</code></pre>
<p>I guess the cause of this error is that the variable <code>last_hidden</code> in line 19 is not contiguous. However, what I don’t understand is that the different results on CPU and GPU.</p>
<p>Why running on CPU wouldn’t raise this error?</p>

username: phan_phan
cooked: <p>No expert, but I suppose CPU and GPU handle memory very differently. <a href="https://stackoverflow.com/a/26999092/4280242" rel="nofollow noopener">Read this for good explanation of contiguity</a>.</p>
<p>Anyway, <strong>try using the <code>.contiguous()</code> method on the tensor which needs it</strong>.</p>
Memory issue in nvidia retinanet

username: EliB
cooked: <p>Hi,</p>
<p>I’m trying to train on my own data. I generated cocco like json with my training and validations sets and trying to run it with default argument values, excecpt --images that i’m excluding since data ois from multiple repos.<br>
Something is loading to much batches to memory and get:</p>
<p>RuntimeError: CUDA out of memory. Tried to allocate 348.35 GiB (GPU 0; 7.93 GiB total capacity; 591.43 MiB already allocated; 6.54 GiB free; 26.57 MiB cached)</p>
<p>How to configure it to adjust batches number ?</p>
<p>Thanks</p>

username: ptrblck
cooked: <p>Could you post a link to the repo you are using?<br>
Most likely there is a flag like <code>--batch_size</code> or <code>-b</code> where you can adjust the batch size.<br>
Alternatively, have a look at the <code>DataLoader</code> initialization in the code and change the <code>batch_size</code> argument there.</p>

username: EliB
cooked: <p>Ok.<br>
I have a config.py file with:</p>
<p>annotations                      =  “/input/train/annotations.json”<br>
val_annotations               = “/input/valid/annotations.json”<br>
backbone                         = ‘ResNet50FPN’<br>
classes                            = 3<br>
model                               = “retinanet_rn50fpn.pth”<br>
fine_tune                           = “/media/Data/ObjectDetector/resnet50-19c8e357.pth”<br>
iters                                  = 100<br>
val_iters                           = 10<br>
lr                                       = 0.0001<br>
resize                               = 516<br>
batch                                = 1<br>
max_size                          = 516</p>
<p>Then in UseExample.py:<br>
import retinanet.config_train as config<br>
import retinanet.main as main<br>
args = <br>
[<br>
“train”            ,<br>
config.model       ,<br>
‘–backbone’       , config.backbone,<br>
‘–annotations’    , config.annotations,<br>
#’–val-annotations’, config.val_annotations,<br>
‘–classes’        , str(config.classes),<br>
‘–resize’         , str(config.resize),<br>
‘–batch’          , str(config.batch),<br>
‘–max-size’       , str(config.max_size),<br>
‘–iters’          , str(config.iters),<br>
‘–lr’             , str(config.lr),<br>
]<br>
main.main(args)</p>

username: ptrblck
cooked: <p>Do you see this OOM error only using your custom dataset or also the original one?<br>
If this error is raised only for your custom dataset, are you using the same image resolution or are you working with bigger images? In the latter case, could you resize your custom images to the same size and try to run the code again?</p>
<p>Since the batch size is already set to 1, you cannot lower is further and would need to save some memory in another part of your training.</p>

username: EliB
cooked: <p>My mistake:</p>
<p>My json cocco like meta file was messed up. Now works wonderfully.</p>
<p>Sorry for bothering you</p>
Training Yolov3 ffrom scratch with a single image

username: Luca_Pamparana
cooked: <p>This is not a Pytorch question per se, although I am using Pytorch for this.</p>
<p>This is not something one wants to do but say I implement the Yolov3 object detection model and I want to train it using a single image which only contains a single object class i.e. I am only interested in this detecting this single object class and on top of it, I have only one labelled instance of it in my training set.</p>
<p>My question is that would such a training scenario even work? I want to train it from scratch i.e. I do not use any pretrained model as a prior input state. I am trying to see if I can get the model to memorize this one instance (which is what should happen as it should overfit massively). However, all the online training code that I have tried cannot really train with this single image as input. What I notice is that the number of detection during training goes to zero after a couple of epochs as training progresses? I wonder if someone more experienced might have some input on this. I was expecting this model with many million parameters to quickly overfit.</p>
Pytorch and tensorboard logging

username: Luca_Pamparana
cooked: <p>I am trying to use pytorch with tensorboard and I run the tensorboard server with the following command:</p>
<p><code>tensorboard --logdir=./runs/</code></p>
<p>Now I am just simulating some fake data as follows:</p>
<pre><code class="lang-auto">import numpy as np
import time
from torch.utils.tensorboard import SummaryWriter

train_writer = SummaryWriter(log_dir="./runs/train/")

for i in range(100):
    v = np.random.randint(10, 100)
    train_writer.add_scalar("loss", v, i)
    #time.sleep(1)

train_writer.close()
</code></pre>
<p>Now if I do not put the sleep method in there, the script finishes and I can see the graph on the tensorboard front end.</p>
<p>I put the <code>sleep</code> in there so that I could see the data arrive in a streaming fashion and be able to see the board updating. However, if I put the <code>sleep</code> in there, no graph ever shows up and all I see is the message that there is no data or it could not find any.</p>
<p>I am not sure if this is a tensorboard issue or if I am doing something wrong in using it from within pytorch.</p>

username: albanD
cooked: <p>Hi,</p>
<p>I am not sure how this works but maybe it waits for enough points to be added before sending it to the server? If you run only for 10 iterations and let it run until the end does it work?</p>

username: Luca_Pamparana
cooked: <p>So, if I add <code>sleep</code>, it does not matter how many iterations I run but it never updates. In fact, even after the program terminates, the chart is not shown. I have to kill tensorboard and restart it for the graph to show.</p>

username: albanD
cooked: <p>Interesting…<br>
So it would be tensorboard not updated even though the data has been written to disk?<br>
<a class="mention" href="/u/smth">@smth</a> will know someone more knowledgeable about this that can find the reason for that ?</p>

username: smth
cooked: <p><a class="mention" href="/u/lanpa">@lanpa</a> <a class="mention" href="/u/orionr">@orionr</a> are the right folks who might know more about this</p>

username: lanpa
cooked: <p>Hi, can you try if <code>train_writer.flush()</code> solve you problem?</p>

username: orionr
cooked: <p><a class="mention" href="/u/luca_pamparana">@Luca_Pamparana</a> are you running TensorBoard without TensorFlow being installed? If so, there was a bug where TB wouldn’t update even if there were updates to an event file. This has been fixed on nightly, so try <code>pip install tb-nightly</code> and see if that solves the issue.</p>

username: Luca_Pamparana
cooked: <p>Thank you. That was it!</p>
What is difference between features and state_dict()

username: Shubham_Sharma
cooked: <p>Hi</p>
<p>What is difference b/w<br>
vgg1 = models.vgg16(pretrained=True).state_dict()</p>
<p>vgg2 = models.vgg16(pretrained=True).features</p>
<p>I have used features  with Sequential and OrderedDict for transfer learning<br>
but I haven’t used state_dict much<br>
as far as I understood I believe state_dict() might be used to load weights and bias independently?</p>

username: tom
cooked: <p>Any <code>nn.Module</code> subclasses’ state_dict method returns a dictionary with the named parameters and buffers. Useful for storing and loading (via <code>load_state_dict</code>). The state dict does not contain any information about how these parameters are used, these are provided by the model.</p>
<p><code>VGG.features</code> is the “feature extractor” (the convolutional part) of the VGG model, a submodule.<br>
So they are really quite different things for different purposes.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Shubham_Sharma
cooked: <p>Thanks I understood the difference</p>
I used libtorch to write resnet18 for training in C++, so how to load resnet18.pth in pytorch to help pre-training

username: CF-chen-feng-CF
cooked: <p>I used libtorch to write resnet18 for training in C++, so how to load resnet18.pth in pytorch to help pre-training；<br>
Cannot import using torch::load(resnet18,"./resnet18.pth")<br>
Resnet18 written in c++ is correct</p>

username: tom
cooked: <p>So no-one answered here for a few days. I think you need to provide some more / more verbose information of what you are trying to do to be able to understand.</p>
<p>Best regards</p>
<p>Thomas</p>
Description of Conv2d groups parameter seems inconsistent with results

username: liam.schoneveld
cooked: <p>The pytorch docs for the <code>groups</code> parameter of nn.Conv2d state that:</p>
<blockquote>
<p>groups controls the connections between inputs and outputs. in_channels and out_channels must both be divisible by groups. For example,</p>
<p>At groups=1, all inputs are convolved to all outputs.</p>
<p>At groups=2, the operation becomes equivalent to having two conv layers side by side, each seeing half the input channels, and producing half the output channels, and both subsequently concatenated.</p>
<p>At groups= in_channels, each input channel is convolved with its own set of filters, of size: in_channels / out_channels</p>
</blockquote>
<p>However, this description seems inconsistent with the behaviour of nn.Conv2d in reality.</p>
<p>for example:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
conv_layer = nn.Conv2d(16, 16, 1, groups=2, bias=False)
conv_layer.weight.shape
</code></pre>
<p>Returns <code>torch.Size([16, 8, 1, 1])</code></p>
<p>But based on my interpretation of :</p>
<blockquote>
<p>At groups=2, the operation becomes equivalent to having two conv layers side by side, each seeing half the input channels, and producing half the output channels, and both subsequently concatenated.</p>
</blockquote>
<p>Shouldn’t there be two weights, each of size <code>[8, 8, 1, 1]</code>?</p>
<p>These inconsistencies carry over to other values for the <code>groups</code> parameter, in my view.</p>
<p>I must be missing something - could someone please clarify?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="liam.schoneveld" data-post="1" data-topic="51253">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/liam.schoneveld/48/11574_2.png" class="avatar"> liam.schoneveld:</div>
<blockquote>
<p>Shouldn’t there be two weights, each of size <code>[8, 8, 1, 1]</code> ?</p>
</blockquote>
</aside>
<p>There are two groups of this size concatenated in dim0.<br>
Have a look at this small example:</p>
<pre data-code-wrap="python"><code class="lang-python">conv_grouped = nn.Conv2d(16, 16, 1, groups=2, bias=False)
output_grouped = conv_grouped(x)
output_grouped.shape

conv1 = nn.Conv2d(8, 8, 1, bias=False)
conv2 = nn.Conv2d(8, 8, 1, bias=False)
with torch.no_grad():
    conv1.weight.copy_(conv_grouped.weight[:8])
    conv2.weight.copy_(conv_grouped.weight[8:])

output1 = conv1(x[:, :8])
output2 = conv2(x[:, 8:])
output_manual = torch.cat((output1, output2), dim=1)

print((output_manual == output_grouped).all())
&gt; tensor(1, dtype=torch.uint8)
</code></pre>
<p>As you can see, the first half of the <code>conv_grouped</code> weights will be applied to the first half of the channels in <code>x</code> and the second half of the weights to the second half of the channels.</p>

username: liam.schoneveld
cooked: <p>Ah thanks so much that’s perfectly clear now!</p>
LSTMCell equivalent in C++ frontend

username: minty99
cooked: <p>I’m converting my model in Python to C++.<br>
I want to use LSTMCell in C++ frontend but I can’t find it. No equivalent for LSTMCell in C++?</p>

username: tom
cooked: <p>There isn’t currently (the nn API is not on par with Python’s – if you want to help change that…). You could just use one-element inputs to LSTM instead.</p>
<p>Best regards</p>
<p>Thomas</p>
Libtorch built from source is 10 times slower than the official binary

username: LJXLJXLJX
cooked: <p>I tried both build_libtorch.py and use cmake myself. But the lib built from source is very slow.<br>
I use the default cmake config.<br>
Is there anyone know the official config ?</p>

username: tom
cooked: <p>Depending on what you are doing, you might be missing accelerator libraries like MKL or CuDNN.<br>
You should get relatively decent diagnostics what is included from cmake and compare that with what you get when you follow the <a href="https://pytorch.org/get-started/locally/#linux-from-source" rel="nofollow noopener">official instructions for building PyTorch</a> from source.<br>
Indeed,  the official libtorch used to indeed just build with python setup.py and then extracted from the Python build (but I am not sure if that is still the case).</p>
<p>Best regards</p>
<p>Tomas</p>