akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Optimize batch processing of variable-length inputs

username: JohnCwok
cooked: <p>I noticed that it is not possible to feed <code>packed_sequences</code> to things like activation functions or linear layers, forcing me to design models having a forward method like :</p>
<pre><code class="lang-auto">    def forward(self, input, lengths, hidden = None):
        input = nn.utils.rnn.pack_padded_sequence(input, lengths, batch_first = True, enforce_sorted = False)
        out, hidden = self.lstm(input,hidden)
        out = nn.utils.rnn.pad_packed_sequence(out, batch_first = True, padding_value= -100)[0]
        out = self.drop_layer(self.sigmoid(out))
        out = self.softmax(self.linear_layer(out))
</code></pre>
<p>Where I have to unpack the <code>packed_sequence</code> directly after the passing through <code>LSTM</code>, and later need to filter it before loss calculation. if I don’t, I received an error message :</p>
<pre><code class="lang-auto">TypeError: sigmoid(): argument 'input' (position 1) must be Tensor, not PackedSequence
</code></pre>
<p>This seem highly inefficient since a lot of the calculations done by the activations functions and linear layers will have to be thrown away afterwards.</p>
<p>Why is that so ? What is the reason preventing us from passing <code>packed_sequences</code> to activation function or linear layers ?</p>
Libtorch adam cuda memory leaks?

username: beichen2012
cooked: <p>libtorch1.2 adam cuda memory leaks, but sgd no leaks<br>
can any one help me?</p>

username: beichen2012
cooked: <p>also occurs in RMSProp</p>

username: ptrblck
cooked: <p>Do you have a reproducible code snippet so that we could have a look?</p>

username: beichen2012
cooked: <p>please refer to :<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://github.com/beichen2012/pytorchAdamMemLeak" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars1.githubusercontent.com/u/10824490?s=400&amp;amp;v=4" class="thumbnail" width="" height="">

<h3><a href="https://github.com/beichen2012/pytorchAdamMemLeak" target="_blank" rel="nofollow noopener">beichen2012/pytorchAdamMemLeak</a></h3>

<p>test libtorch train step for adam(RMSProp) memory leaks in CUDA - beichen2012/pytorchAdamMemLeak</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

username: beichen2012
cooked: <p>there is no leaks in CPU.<br>
and in GPU,  2M leaks every time .</p>

username: beichen2012
cooked: <p>please also refter to :<br>
<aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/25303" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <a href="https://github.com/beichen2012" rel="nofollow noopener">
<img src="https://avatars1.githubusercontent.com/u/10824490?v=2&amp;s=96" class="thumbnail onebox-avatar" width="96" height="96">
</a>

<h4><a href="https://github.com/pytorch/pytorch/issues/25303" target="_blank" rel="nofollow noopener">Issue: libtorch(c++) Adam and RMSProp optimizer memory leaks in CUDA </a></h4>

<div class="date" style="margin-top:10px;">
	<div class="user" style="margin-top:10px;">
	opened by <a href="https://github.com/beichen2012" target="_blank" rel="nofollow noopener">beichen2012</a>
	on <a href="https://github.com/pytorch/pytorch/issues/25303" target="_blank" rel="nofollow noopener">2019-08-28</a>
	</div>
	<div class="user">
	</div>
</div>

<pre class="content" style="white-space: pre-wrap;">🐛 Bug
libtorch(c++) Adam and RMSProp optimizer memory leaks in CUDA
To Reproduce
libtorch1.2, cuda10,win10, vs2015, win64
Steps to reproduce the behavior:
please refer to the...</pre>

<div class="labels">
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

username: ptrblck
cooked: <p>Thanks for the code and for opening an issue.<br>
For the same of completeness: the issue is tracked <a href="https://github.com/pytorch/pytorch/issues/25303" rel="nofollow noopener">here</a>.</p>
[SOLVED] NLLLoss not zero for indentical input and target

username: JohnCwok
cooked: <p>I noticed a strange thing with the following input tensor (the 1 is at position <code>62</code>) :</p>
<pre><code class="lang-auto">tensor([[0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0.]])
</code></pre>
<p>and target :</p>
<pre><code class="lang-auto">tensor([62])
</code></pre>
<p>the following commands gives the following output :</p>
<pre><code class="lang-auto">crit = nn.NLLLoss()
soft = nn.Softmax(dim = 1)

crit(input[0,:].unsqueeze(0),target[0].unsqueeze(0))
Out[120]: tensor(-1.)

crit(soft(input[0].unsqueeze(0)),target[0].unsqueeze(0))
Out[135]: tensor(-0.0270)
</code></pre>
<p>This is not zero but it should be… why is that so ? The unsqueeze are here so that NLLLoss is satisfied by input dimensions,</p>

username: SoucheChapich
cooked: <p>Hello,</p>
<p>I think, NLLLoss() works with log probabilities so you should use a LogSoftmax() instead of Softmax().</p>

username: SoucheChapich
cooked: <p>But from the documentation: <a href="https://pytorch.org/docs/stable/nn.html" rel="nofollow noopener">https://pytorch.org/docs/stable/nn.html</a> you can see that the LogSoftmax() will compute LogSoftmax(xi​)=log(exp(xi​)​ / ∑_j ​exp(xj​)). So you are summing 61 times exp(0) = 1  + exp(1) = e in your case. So the behavior is normal. So basically, if you want your loss to be 0, the number at the 62th item should be really high compared to the other ones. Try to put 20 instead of 1 and you will have a loss with 0 value.</p>
<p>Just tell me if I was not clear enough.</p>

username: JohnCwok
cooked: <p>Changing the value of 1 to a higher value solved the issue, thanks for the help. So I guess I can safely implement what I had in mind, the loss will behave as expected.</p>
Check initialization method of Conv2D

username: Javier_Naranjo
cooked: <p>Hello community,</p>
<p>I have implemented a CNN using nn.Sequential and then I have applied a initialization method using</p>
<p><code>net.apply(init_method)</code></p>
<p>I would like to check if the layers have been initializated correctly. Is there any way to check it as a property?</p>
<p>Thanks in advance</p>

username: spanev
cooked: <p>Hi <a class="mention" href="/u/javier_naranjo">@Javier_Naranjo</a></p>
<p>I don’t think there is any generic way of testing it, since it depends on your <code>init_method</code>.<br>
What you can do it looping over the parameters and checking some stats:</p>
<pre><code class="lang-auto">for name, p in net.named_parameters():
   check_method(name, p)
</code></pre>
<p>This check method could be looking at e.g. <code>p.mean()</code> or <code>p.std()</code> and you could draw a conclusion from it.</p>
<p>Hoping this answers somehow your question.</p>

username: Javier_Naranjo
cooked: <p>Inside init_method I am only calling a <code>nn.init</code></p>
<p>My question was if there is any attribute where I can find the initialization method I have used.</p>
<p>Anyway, if there is not any chance, I would do that you mention.</p>
<p>Thanks for the reply</p>

username: spanev
cooked: <p><code>nn.init</code> functions only run simple in-place init operations (without autograd) on the Tensor that represent the params. As you can see <a href="https://github.com/pytorch/pytorch/blob/febcb3b7b37d0ce8a8fdfd1165b37bed2536c118/torch/nn/init.py#L19" rel="nofollow noopener">here</a> (called <a href="https://github.com/pytorch/pytorch/blob/febcb3b7b37d0ce8a8fdfd1165b37bed2536c118/torch/nn/init.py#L247" rel="nofollow noopener">there</a>).<br>
These operations don’t have any side effects except the change of values in the param Tensor.</p>

username: Javier_Naranjo
cooked: <p>Thanks for your help Serge!</p>
Why there is no graph when I use tensorboard to visualize my model

username: crissallan
cooked: <p>I want to use this code snippet to visuallize my model, but find nothing on local host:<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/1/1e3ee69730b2b472ae30fd85212f7ac7f3df7674.png" data-download-href="https://discuss.pytorch.org/uploads/default/1e3ee69730b2b472ae30fd85212f7ac7f3df7674" title="49"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/1/1e3ee69730b2b472ae30fd85212f7ac7f3df7674_2_690x372.png" alt="49" data-base62-sha1="4jz5tzb66mRzku5FbwI8PUpd4z2" width="690" height="372" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/1/1e3ee69730b2b472ae30fd85212f7ac7f3df7674_2_690x372.png, https://discuss.pytorch.org/uploads/default/optimized/2X/1/1e3ee69730b2b472ae30fd85212f7ac7f3df7674_2_1035x558.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/1/1e3ee69730b2b472ae30fd85212f7ac7f3df7674_2_1380x744.png 2x" data-dominant-color="F9F3EE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">49</span><span class="informations">2876×1552 214 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> 6060<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/6/69a04c75b948e6d0d408159673bb67635670b5ba.png" data-download-href="https://discuss.pytorch.org/uploads/default/69a04c75b948e6d0d408159673bb67635670b5ba" title="29"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/6/69a04c75b948e6d0d408159673bb67635670b5ba_2_690x397.png" alt="29" data-base62-sha1="f4pzvwnZE5Gc6MpTtEISRWVZ5ZU" width="690" height="397" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/6/69a04c75b948e6d0d408159673bb67635670b5ba_2_690x397.png, https://discuss.pytorch.org/uploads/default/original/2X/6/69a04c75b948e6d0d408159673bb67635670b5ba.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/6/69a04c75b948e6d0d408159673bb67635670b5ba.png 2x" data-dominant-color="353535"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">29</span><span class="informations">914×526 60 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>Seems to be related to <a href="https://discuss.pytorch.org/t/tensorboard-not-displaying-graph/54497/2">this issue</a>.</p>
<p>EDIT: it seems to be tracked <a href="https://github.com/pytorch/pytorch/issues/24157" rel="nofollow noopener">here</a>.</p>
Calculating outer subtraction for a batch

username: Bhavya_Bhatt
cooked: <p>I know we can calculate outer subtraction using broadcasting as given in<br>
<aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stackoverflow.com/questions/52780559/outer-sum-etc-in-pytorch" target="_blank" rel="nofollow noopener">stackoverflow.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/3719101/iago-lito" target="_blank" rel="nofollow noopener">
    <img alt="iago-lito" src="https://i.stack.imgur.com/Z4L8i.jpg?s=128&amp;g=1" class="thumbnail onebox-avatar" width="60" height="60">
  </a>
<h4>
  <a href="https://stackoverflow.com/questions/52780559/outer-sum-etc-in-pytorch" target="_blank" rel="nofollow noopener">Outer sum, etc. in pytorch</a>
</h4>

<div class="tags">
  <strong>python, optimization, pytorch, torch</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/3719101/iago-lito" target="_blank" rel="nofollow noopener">
    iago-lito
  </a>
  on <a href="https://stackoverflow.com/questions/52780559/outer-sum-etc-in-pytorch" target="_blank" rel="nofollow noopener">01:26PM - 12 Oct 18 UTC</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
But how can I do it for batches where first dimension is batch size and then we have second dimension to be the vectors for which we want to calculate outer subtraction between each vector in the first tensor with every other vector in the second tensor.</p>

username: yuval
cooked: <p>See <a href="https://stackoverflow.com/questions/55739993/pytorch-batch-outer-addition" rel="nofollow noopener">https://stackoverflow.com/questions/55739993/pytorch-batch-outer-addition</a></p>
Difference between view, reshape and permute

username: Rafael_R
cooked: <p>Are these operations fundamentally different?</p>

username: ptrblck
cooked: <p><code>reshape</code> tries to return a <code>view</code> if possible, otherwise copies to data to a contiguous tensor and returns the view on it. From the <a href="https://pytorch.org/docs/stable/torch.html#torch.reshape">docs</a>:</p>
<blockquote>
<p>Returns a tensor with the same data and number of elements as <code>input</code> , but with the specified shape. When possible, the returned tensor will be a view of <code>input</code> . Otherwise, it will be a copy. Contiguous inputs and inputs with compatible strides can be reshaped without copying, but you should not depend on the copying vs. viewing behavior.<br>
See <a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor.view"> <code>torch.Tensor.view()</code> </a> on when it is possible to return a view.<br>
A single dimension may be -1, in which case it’s inferred from the remaining dimensions and the number of elements in <code>input</code> .</p>
</blockquote>
<p>Have a look at this example to demonstrate this behavior:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.arange(4*10*2).view(4, 10, 2)
y = x.permute(2, 0, 1)

# View works on contiguous tensors
print(x.is_contiguous())
print(x.view(-1))

# Reshape works on non-contugous tensors (contiguous() + view)
print(y.is_contiguous())
try: 
    print(y.view(-1))
except RuntimeError as e:
    print(e)
print(y.reshape(-1))
print(y.contiguous().view(-1))
</code></pre>
<p><code>permute</code> is quite different to <code>view</code> and <code>reshape</code>:</p>
<pre data-code-wrap="python"><code class="lang-python"># View vs. permute
x = torch.arange(2*4).view(2, 4)
print(x.view(4, 2))
&gt; tensor([[0, 1],
          [2, 3],
          [4, 5],
          [6, 7]])
print(x.permute(1, 0))
&gt; tensor([[0, 4],
          [1, 5],
          [2, 6],
          [3, 7]])
</code></pre>

username: Rafael_R
cooked: <ol>
<li>so when training a model, is it best to use view?</li>
<li>Is using reshape a source of possible bugs in terms of gradient flow?</li>
</ol>
<p>Anything to be careful otherwise as well?</p>
<p>Thanks!</p>

username: ptrblck
cooked: <ol>
<li>
<p>Not necessarily. The usage of <code>view</code> and <code>reshape</code> does not depend on training / not-training.<br>
I personally use <code>view</code> whenever possible and add a <code>contiguous</code> call to it, if necessary. This will make sure I see, where a copy is done in my code. <code>reshape</code> on the other hand does this automatically, so your code might look cleaner.</p>
</li>
<li>
<p>No, it should not be a bug regarding the gradient flow. However, as shown with the <code>reshape</code> vs. <code>permute</code> example, the wrong operator might of course cause problems in your training. E.g. if you would like to swap some axes of an image tensor from NCHW to NHWC, you should use <code>permute</code>.</p>
</li>
</ol>
Torchvision.io inconsistent with the rest of the module

username: Cristian_23
cooked: <p>I am trying to get images directly from a video using torchvision.io.read_video. I noticed this function reads images in the form [H,W,C] while everything else in torchvision expects [C,H,W]. Why the inconsistency?</p>
<p>Is there an inexpensive way to get the frames from torchvision.io.read_video in [C,H,W] form?</p>
Why CUDA out of memory during training?

username: ilyes
cooked: <p>Hello,</p>
<p>Here’s a simple code:</p>
<pre><code class="lang-auto">class Identity(nn.Module):
    def __init__(self):
        super().__init__()

    def forward(self, x):
        return x
    
class CNNLSTM(nn.Module):
    def __init__(self):
        super(CNNLSTM, self).__init__()
        self.BackBone = models.resnet18(pretrained= True)
        num_ftrs = self.BackBone.fc.in_features
        self.BackBone.fc = Identity()
        self.lstm = nn.LSTM(512, 512, batch_first  = True)
        self.fc = nn.Linear(512, 1)
    def forward(self, video):
        batch_size, time_steps, C, H, W = video.size()
        c_in = video.view(batch_size * time_steps, C, H, W)
        c_out = self.BackBone(c_in)
        r_in = c_out.view(batch_size, time_steps, -1)
        r_out, _  = self.lstm(r_in)
        output = torch.sigmoid(self.fc(r_out[:, -1, :]))
        return output.squeeze()
</code></pre>
<p>If I run the training loop without actually using the model, means only fetching the batches for a number of epochs my GPU uses 1.1 GB out of 4.0 GB.<br>
means I only do this:</p>
<pre><code class="lang-auto">for epoch in range(num_epochs):
        print('Epoch {}/{}'.format(epoch+1, num_epochs))
        for i, (inputs, labels) in enumerate(dataloader):
            inputs = inputs.to(device, non_blocking=True)
            labels = labels.to(device , non_blocking=True)

</code></pre>
<p>If I call my model however on the batch, I get CUDA out of memory. Means if I only add these to my loop:</p>
<pre><code class="lang-auto">            optimizer.zero_grad()
            outputs = model(inputs)
</code></pre>
<p>where does the 3 GB go! do the gradients occupy memory. Because I thought once you call the model in:</p>
<pre><code class="lang-auto">model = CNNLSTM()
model.to(device)
</code></pre>
<p>The graphs and the gradients have already allocated their memory?<br>
Thank you</p>

username: spanev
cooked: <p>When you actually execute the model, the computation graph gets built dynamically (recording all the operations).</p>
<p>What (I think…) takes most of the memory is the outputs of some operations (the edges of the graph). Those will be needed by autograd to compute the gradients so are kept in memory until you call <code>outputs.backward()</code>.</p>
<p>Please find more details <a href="https://pytorch.org/docs/stable/notes/autograd.html#how-autograd-encodes-the-history" rel="nofollow noopener">here</a> and autograd concept better explained than by me.</p>

username: ilyes
cooked: <p>I will check the link, Thank you!</p>
Can CosineSimilarity do multiple calculation at one time?

username: 888
cooked: <p>Hi all,</p>
<p>I have two tensors with shape [B1, N] and [B2, N], N is the vector length.</p>
<p>I want to get a similarity metric with shape [B1, B2] with the current API.</p>
<p>Can I do that at once? I found that it always rises shape mismatching error.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/5/57e2ee6fa68369745392a7fedf488800a7b0e403.png" data-download-href="https://discuss.pytorch.org/uploads/default/57e2ee6fa68369745392a7fedf488800a7b0e403" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/5/57e2ee6fa68369745392a7fedf488800a7b0e403_2_690x379.png" alt="image" data-base62-sha1="cxtJLBTeVhaXwIuWA6CiNDifOdd" width="690" height="379" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/5/57e2ee6fa68369745392a7fedf488800a7b0e403_2_690x379.png, https://discuss.pytorch.org/uploads/default/original/2X/5/57e2ee6fa68369745392a7fedf488800a7b0e403.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/5/57e2ee6fa68369745392a7fedf488800a7b0e403.png 2x" data-dominant-color="0B0C0A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">962×529 61.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: Sunshine352
cooked: <p>YEs, you can by the way below.<br>
Assume the shape of x1 and x2 is  [B1, N] and  [B2, N], then cosine similarity is computed below.<br>
x1_norm = F.normalize(x1, p=2, dim=1)<br>
x2_norm = F.normalize(x2, p=2, dim=1)<br>
cosine similarity = x1_norm * x2_norm.t()</p>

username: mmisiur
cooked: <p>Yes, but you have to unsqueeze and swap shape, so the [B1, N] becomes [B1, N, 1] and [B2, N] becomes [1, N, B2]:</p>
<pre><code class="lang-auto">t1 = torch.randn((17,1024,1))
t2 = torch.randn((1,1024,512))
s1 = torch.nn.functional.cosine_similarity(t1,t2)
print(s1.shape)
</code></pre>
<p>gives</p>
<pre><code class="lang-auto">torch.Size([17, 512])
</code></pre>
Can one lstm model capture multiple patterns?

username: pahome
cooked: <p>I am trying to use an LSTM to predict  daily usage for users. I have data for 30 days.<br>
Based on business knowledge I know users divide roughly into different categories. E.g. daily users would have a non-zero usage almost every day, weekly users have one or two days of non-zero usage every 7 days and monthly users might have a couple of days with non-zero usage per 30 days.</p>
<p>Samples where every column is one day and each row is <strong>access usage</strong> for one user.</p>
<pre><code class="lang-auto">User 1: 50,  80, 33,  19, 30, 15, ...
User 2:  0,  21,  13, 30, 0, 5,  0,  0,  55, 28,  0,  19,  0, ...
User 3: 11,  2,  11,  56, .....
</code></pre>
<p>From above, User1,3 maybe a daily user, User2 maybe a weekly user.</p>
<p>If I only can get info such as <strong>user name</strong> and <strong>access usage</strong>.</p>
<p>Can a single lstm model capture this different types of patterns of users?<br>
The goal is to predict the daily usage for next 10 days of each users.</p>
<p>Why I ask is because I tried 100 epochs with learning rate 0.0001, but error still failed.<br>
The prediction result always look like the same even give another input of user name.</p>
Indexing with a tuple aliases, but with a python list copies

username: drevicko
cooked: <p>I noticed what I find to be surprising behaviour: if I index a tensor with a python tuple, I get an alias of the indexed element, but if I index with a python list, I get a copy:</p>
<blockquote>
<p>t = torch.rand(3,5)<br>
print(t[1,2].data_ptr())<br>
idx = (1,2)<br>
print(t[idx].data_ptr())<br>
idx = [1,2]<br>
print(t[idx].data_ptr())</p>
</blockquote>
<p>Output:</p>
<pre><code>94484139998412
94484139998412 
94484140672144
</code></pre>
<p>Is this expected behaviour? I gather that indexing with a tuple is identical to directly indexing an element, so returning a reference/alias makes sense.</p>
<p>Am I to understand that a python list is converted to a tensor, and that tensor indexing with a non-zero-dim tensor results in a copy? If so, <a href="https://github.com/pytorch/pytorch/issues/6217" rel="noopener nofollow ugc">this discussion</a> around indexing with zero-dim tensors seems relevant here.</p>
<p>There’s also <a href="https://discuss.pytorch.org/t/indexing-tensor-using-array-python-list-vs-torch-tensor/41591">this post</a> on a possibly related confusion.</p>

username: andreaskoepf
cooked: <p>By using a list to index a tensor you can select elements of a dimension in arbitrary order. Due to the internal representation of a tensor over its underlying continuous storage it is in general not possible to create a new tensor that is just a view into the old one and shares its storage object. In contrast by using a tuple you can specify only single elements along each dimension - each tuple entry is responsible for a different dimension.</p>
<p>e.g. lets say you want to index the diagonal elements in reverse:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; x = torch.rand(3,3)
&gt;&gt;&gt; x
tensor([[0.1899, 0.9408, 0.0889],
        [0.4863, 0.5366, 0.1633],
        [0.8910, 0.4463, 0.2007]])
&gt;&gt;&gt; x[[-1,-2,-3],[-1,-2,-3]]
tensor([0.2007, 0.5366, 0.1899])
</code></pre>

username: drevicko
cooked: <p>Oh, I get it. A tuple indexes an element, a list provides multiple indexes in the first dimension of <code>t</code>.</p>
<pre><code class="lang-auto">In [31]: t = torch.rand(3,5) 
    ...: print(t[1,2])
    ...: print(t[1,2].data_ptr()) 
tensor(0.1442)                                                                                                                                                                                                          
140485274866972

In [32]: idx = (1,2) 
    ...: print(t[idx]) 
    ...: print(t[idx].data_ptr()) 
tensor(0.1442)
140485274866972
In [33]: idx = [1,2] 
    ...: print(t[idx])   
    ...: print(t[idx].data_ptr()) 
tensor([[0.8098, 0.4710, 0.1442, 0.5391, 0.1699],
        [0.7292, 0.6585, 0.7074, 0.4800, 0.6104]])
140485233816192
</code></pre>
MultiProcessing with PyTorch

username: Soniya
cooked: <p>I want to use the concept of multiprocessing with PyTorch. Are there any basic tutorials for this?</p>

username: fanq15
cooked: <p><code>nn.parallel.DataParallel</code><br>
<a href="https://discuss.pytorch.org/t/is-the-loss-function-paralleled-when-using-dataparallel/3346">https://discuss.pytorch.org/t/is-the-loss-function-paralleled-when-using-dataparallel/3346</a></p>

username: smth
cooked: <p><a href="http://pytorch.org/docs/notes/multiprocessing.html" class="onebox" target="_blank">http://pytorch.org/docs/notes/multiprocessing.html</a><br>
also <a href="https://docs.python.org/2/library/multiprocessing.html">https://docs.python.org/2/library/multiprocessing.html</a></p>

username: Soniya
cooked: <p>Thanks Smth. The second link is more useful for me.</p>
<p>Thanks <img src="//discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: larabrian
cooked: <p>Check with Python… <a href="http://net-informations.com/python/iq/multi.htm" rel="nofollow noopener">threading Vs. Multiprocessing</a></p>
Looking for sparse matrix solvers in pytorch

username: Rina_Buoy
cooked: <p>Hello,</p>
<p>I am looking for sparse matrix solvers (Ax=b) like BiCGStab in Pytorch. I understand that we can use one of the optimisers in PyTorch to that, but I am not sure how well its performance, compared with iterative solvers like BiCGStab.</p>
<p>Can anyone help ?</p>
Dealing with multiple inputs for onnx export

username: kl_divergence
cooked: <p>My model takes multiple inputs (9 tensors), how do I pass it as one input in the following form:</p>
<pre><code class="lang-auto">torch.onnx.export(model,inputs,'model.onnx')
</code></pre>
<p>I’ve tried putting all the tensors in the list and passing it as input.<br>
<code>TypeError: forward() missing 8 required positional argument</code>. What can be a work around for this ?</p>

username: kl_divergence
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Could you please suggest some workaround ?</p>

username: ptrblck
cooked: <p>I’m not really experienced in ONNX, so that unfortunately I cannot really help.</p>
<p>However, could you just change the <code>forward</code> method of your model to take a list instead of 9 arguments?</p>

username: kl_divergence
cooked: <p>My forward method takes 9 arguments only. But in this case, they (as per docs) assume that only one tensor is passed to model. But what about models which take multiple tensors as input ?</p>

username: gusta
cooked: <p>I have a similar problem but looks simpler taking only two arguments inputs.<br>
For your case, from other forum, the solution is to wrap your input as tuple e.g. instead of</p>
<pre><code class="lang-auto">onnx.export(model, inputs, ...)
</code></pre>
<p>use this:</p>
<pre><code class="lang-auto">onnx.export(model, (inputs,), ...)
</code></pre>
Setting g in weight normalization to 1

username: Maria_Fernanda_De_La
cooked: <p>Hello, I want to do some experiments with weight normalization and I was wondering if it would be possible to set the g in weight normalization as 1 or a different function of time in PyTorch.</p>
<p>Thank you in advance!</p>

username: Maria_Fernanda_De_La
cooked: <p>I believe this might be working: net.fc1.weight_g = nn.Parameter(torch.Tensor(1))</p>
Generating the class activation maps

username: vmirly1
cooked: <p>I have trained a CNN model with softmax classification layer. Now, I want to get the class activation maps (CAM) from these trained model on some test-samples. I have found a code that does this using Keras, but cannot do the same thing in PyTorch.</p>
<p>In this Keras code, they compute the gradients of the predicted output with respect to the last convolutional layer. So, I wonder how I should compute these gradients in PyTorch.</p>
<p>Here is the full Keras code:</p>
<pre><code class="lang-auto">    preds = model.predict(image)
    class_idx = np.argmax(preds[0])
    class_output = model.output[:, class_idx]
    last_conv_layer = model.get_layer("mixed10")

    grads = K.gradients(class_output, last_conv_layer.output)[0]
    pooled_grads = K.mean(grads, axis=(0, 1, 2))
    iterate = K.function([model.input], [pooled_grads, last_conv_layer.output[0]])
    pooled_grads_value, conv_layer_output_value = iterate([image])

    for i in range(2048):
        conv_layer_output_value[:, :, i] *= pooled_grads_value[i]

    heatmap = np.mean(conv_layer_output_value, axis = -1)
    heatmap = np.maximum(heatmap, 0)
    heatmap /= np.max(heatmap)
</code></pre>
<p>Any help is appreciate!</p>

username: rasbt
cooked: <p>How about defining a hook? Would that solve your problem? E.g., something like</p>
<pre><code class="lang-python">class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
 
        self.gradients = []

        ## your layers
        self.some_conv_layer.register_backward_hook(self.save_gradients)
        
    def forward(self, x):
        ## your forward pass
        return F.softmax(logits)

    def save_gradients(self, module, grad_input, grad_output):
        self.gradients.append(grad_output)

model = MyModel()
output = model(x)
output.backward()

print(model.gradients)
</code></pre>

username: vmirly1
cooked: <p>Thank you very much! I think that solved the problem, and with that I should be able to get the activation maps.</p>

username: Ziri_Ziri
cooked: <p>Hi everyone , Is there a way to achieve the same using c++ frontend ?<br>
Thank you.</p>
How to solve the loss become nan because of using torch.log()

username: crissallan
cooked: <p>I met a ‘nan’ loss problem because of introducing a torch.log(t) operation in the forward pass. when I removed the log operation, things work fine. I think it’s because of the tensor t contains very small or zero number. does that mean if the forward process produces some ‘nan’ numbers, the loss will must be ‘nan’ number. e.g. if I put a ‘nan’ tensor into a nn.linear layer, what will be the output of this linear layer?</p>
<p>However, if I do the log operation using numpy and turn the data being logged to a tensor, things still work fine, no nan error in the loss.</p>
<p>I’d like to know how the extremely small numbers are treated in back propagation in Pytorch. And what is the grad of a ‘nan’  loss.<br>
Thank you so much!</p>

username: alekhka
cooked: <p>This is very likely because the input is a negative number.</p>
<p>Since logarithmic function has the domain x&gt;0, you have to ensure that the input is non-negative and non-zero.</p>
<p>I would use a non-linearity like ReLU or sigmoid to ensure non-negativity and then add a small ‘epsilon’ to ensure non-zero:</p>
<pre><code class="lang-auto">eps=1e-7
t = F.relu(t)
t = torch.log(t +eps)
</code></pre>
<p>All operations on nan result in nan, so everything will become nan quickly.</p>
Classification without onehot encoding

username: 111137
cooked: <p>Hi,</p>
<p>Can we code a classification without onehot encoding, for example;</p>
<pre><code class="lang-python">x, y = dataloader() ##Load input x and label y(integer scalar)
out = model()
z = round(out) ##Rounding scalar "out" to integer scalar "z"
loss = criteria(z, y)
</code></pre>
<p>If this is possible, how we treat back prop on the rounding?</p>

username: tom
cooked: <ul>
<li>You cannot backprop through round.</li>
<li>You  could take the (square, or something) difference of z and y, i.e. solving a regression problem.</li>
<li>Regression on class numbers doesn’t work particularly well for classification in general. (We would not be making the distinction if it did.)</li>
</ul>
<p>Best regards</p>
<p>Thomas</p>

username: KFrank
cooked: <p>Hi NaN!</p>
<aside class="quote no-group quote-modified" data-username="111137" data-post="1" data-topic="54424" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/111137/48/15236_2.png" class="avatar"> 111137:</div>
<blockquote>
<p>Can we code a classification without onehot encoding,<br>
…</p>
</blockquote>
</aside>
<p>If you’re asking <em>specifically</em> about whether we can avoid<br>
one-hot encoding the target labels, the answer is yes.</p>
<p>Let’s say we have a classification problem with 5 classes.<br>
The most common approach (probably) is to build a network<br>
whose output (for a single sample) is a length-5 vector (of<br>
real numbers, not integers).  But the label (for a single sample)<br>
is a single integer class label (so not one-hot encoded).</p>
<p>Then we run the output and label through, for example,<br>
<code>nn.CrossEntropyLoss</code> as the loss criterion.</p>
<p>Again, in this scheme, the output of your network is a vector of<br>
length number-of-classes, but your labels are single numbers.</p>
<p>Best.</p>
<p>K. Frank</p>

username: 111137
cooked: <p><a class="mention" href="/u/kfrank">@KFrank</a> -san,</p>
<p>Thank you, yes I want to do is so.<br>
Just, making output layer as single out (float scalar) and doing train with the value and label (as float scalar). After that cap the rounding to the output to make precise integer number at inference service.</p>
Fixed line layer

username: slavavs
cooked: <p>Can I make such a linear layer in which weights will not change?<br>
I want to compress the output from 10 neurons to 3, but so that the linear layer is not trained.</p>
<pre><code class="lang-auto">self.out = nn.Linear(10, 3)
</code></pre>

username: spanev
cooked: <p>You can do:</p>
<pre><code class="lang-auto">for p in model.out.parameters():
    p.requires_grad = False
</code></pre>

username: ptrblck
cooked: <p><code>eval()</code> won’t work in this case, as it switches the behavior of certain layers like batchnorm and dropout layers.<br>
Set the <code>requires_grad</code> attribute to <code>False</code> for <code>self.out.weight</code> and <code>self.out.bias</code> to fix these parameters.</p>
<p>EDIT: I was too slow, as <a class="mention" href="/u/spanev">@spanev</a> already added the right approach <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: slavavs
cooked: <p>Did you mean that?</p>
<pre><code class="lang-auto">def forward(self, x):
        out = self.fc1(x) #Linear 1
        out = self.relu1(out)
        out = self.bn1(out)
        
        out = self.fc1(out) #Linear 2
        self.out.weight.requires_grad = False
        self.out.bias.requires_grad = False
        
        return out
</code></pre>

username: ptrblck
cooked: <p>If you don’t want to train these parameters at all, follow <a class="mention" href="/u/spanev">@spanev</a>’s approach and set the <code>requires _grad</code> attribute to <code>False</code> after creating an instance of the model.</p>

username: slavavs
cooked: <p>I want to freeze only one linear layer. I simply gave a simplified example.</p>

username: ptrblck
cooked: <p><a class="mention" href="/u/spanev">@spanev</a>’s example only freezes the parameters of <code>model.out</code>, which would refer to your linear layer.</p>

username: slavavs
cooked: <p>I can’t figure out where in the code I need to insert this.</p>
<pre><code class="lang-auto">AttributeError: 'model' object has no attribute 'out'
</code></pre>
<p>Sorry, got it</p>
<p>…<br>
I wonder how to check if the weights are changing)))</p>

username: spanev
cooked: <p>Can you share the code of the model or at least the concerned part of the model?<br>
If <code>out</code> is nested in another layer or class you would have to do something like <code>model.submodule_name.out</code></p>
Mean of a Padded Sequence

username: Walsh
cooked: <p>Given a Tensor containing a set of padded sequences with shape B x P + 1, where the + 1 is a column containing the lens of the sequences, how can I find the mean of each sequence? Thanks in advance.</p>
<p>edit - Please note, that the padded elements are non-zero since they result from a previous embedding step</p>

username: zhangguanheng66
cooked: <p>You may consider <a href="https://pytorch.org/docs/master/nn.html?highlight=embeddingbag#torch.nn.EmbeddingBag" rel="nofollow noopener">EmbeddingBag</a>. It will calculate the mean value without padding.</p>

username: Walsh
cooked: <p>Thanks <a href="/u/zhangguanheng66">Guanheng</a>, unfortunately my sequences are not fixed length so it seems I cannot use the EmbeddingBag approach. Do you have some other ideas?</p>
<ul>
<li>Edit 1: This is there error I’m seeing that lead me to this assumption:<br>
“ValueError: if input is 2D, then offsets has to be None, as input is treated is a mini-batch of fixed length sequences”</li>
<li>Edit 2: I was incorrect, you can use EmbeddingBag with variable length sequences. One needs to first flatten the sequences (to avoid the above 2D error) and provide offsets to the positions of the tensors</li>
</ul>
CuDNN issues with backprop for sensitivity analysis

username: Steve_Draper
cooked: <p>I need to back-propagate some gradients at eval time (to perform sensitivity analysis for explanation generation).  However, CuDNN does not support back-prop except in train mode.</p>
<p>This creates an awkward dilemma  - if I try to stick with running the model in eval mode then I can only do the explanation generation on CPU.  Conversely, if I run the model in train mode I have to worry about the <em>other</em> semantic impacts that has - in particular the fact that it will be interpreted by <code>Dropout</code> modules.</p>
<p>For now I have (not very nice) work-arounds such as:</p>
<ol>
<li>Hack my model loader to set all dropout values to 0 on load, since I (happen to) know I’m only going to be using it for prediction</li>
<li>In the master <code>Module</code> class for my model, add a call (say <code>allow_back_prop</code>) that calls <code>train()</code> on the RNN sub-modules only, so that CuDNN sees its components running in train mode even though the overall model is in eval mode (I <em>think</em> this will work, right?)</li>
</ol>
<p>However, what I really need (absent a relaxation in the constraint CuDNN is imposing) is for PyTorch to separate different aspect of <code>train</code> vs <code>eval</code> semantics with either separate methods or extra optional parameters to the <code>train</code> method.</p>
<p>If anyone has any other thoughts/recommendations on this I’d love to discuss.</p>
RuntimeError: size mismatch, m1: [1 x 1000000], m2: [784 x 125] at /pytorch/aten/src/TH/generic/THTensorMath.cpp:961

username: Nisheeth_Jaiswal
cooked: <p>import requests<br>
from PIL import Image</p>
<p>url = ‘<a href="https://images.homedepot-static.com/productImages/007164ea-d47e-4f66-8d8c-fd9f621984a2/svn/architectural-mailboxes-house-letters-numbers-3585b-5-64_1000.jpg" rel="nofollow noopener">https://images.homedepot-static.com/productImages/007164ea-d47e-4f66-8d8c-fd9f621984a2/svn/architectural-mailboxes-house-letters-numbers-3585b-5-64_1000.jpg</a>’<br>
response = requests.get(url, stream = True)<br>
img = Image.open(response.raw)<br>
plt.imshow(img)</p>
<p>img = PIL.ImageOps.invert(img)<br>
img = img.convert(‘1’)<br>
img = transform(img)<br>
plt.imshow(im_convert(img))</p>
<p>img = img.view(img.shape[0], -1)<br>
output = model(img)<br>
_, pred = torch.max(output, 1)<br>
print(pred.item())</p>
<p><strong>I’m getting this error for the above code</strong></p>
<p>RuntimeError: size mismatch, m1: [1 x 1000000], m2: [784 x 125] at /pytorch/aten/src/TH/generic/THTensorMath.cpp:961</p>
<p>Please someone can help me resolve this issue!!!</p>

username: Nisheeth_Jaiswal
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> can you suggest something in here hoe to solve the issue!</p>
NLLLoss not correctly ignoring padding token?

username: JohnCwok
cooked: <p>Hi !</p>
<p>I am trying to train a LSTM-based network with variable-lengthed inputs using padded sequences.<br>
In order for the training to be batched processed as well I flatted the padded output as well as the padded target before feeding it to <code>NLLLoss()</code>, which is suppose to ignore field values of <code>-100</code> as default behavior (according to the doc).</p>
<p>I did a test where I fed the padded input and padded corresponding target to <code>NLLLoss</code> expecting to obtain <code>0</code> since these two tensor are containing the same informations but the value that I get is <code>-0.0514</code>.<br>
Is this a numerial error or a bug in my code ?</p>
<p>Here is my test :</p>
<pre><code class="lang-auto">crit = nn.NLLLoss()
input = inputTensor.view(-1, inputTensor.size(2))
target = target.view(-1)

crit(input, target)
Out[91]: tensor(-0.0514)
</code></pre>
<p>4 being my batch size, input and target are, before flattening :</p>
<pre><code class="lang-auto">input.size()
Out[87]: torch.Size([4, 1631, 99])
target.size()
Out[88]: torch.Size([4, 1631])
</code></pre>
<p>and after flattening :</p>
<pre><code class="lang-auto">input.size()
Out[89]: torch.Size([6524, 99])
target.size()
Out[90]: torch.Size([6524])
</code></pre>
Libtorch with ROS

username: sanketgujar
cooked: <p>I am having a hard time running pytorch model with ROS in C++. The torchlib breaks the ROS when I try to include it in my package cmakelist. I included the moden ABI by “add_compile_options(-std=c++11 -D_GLIBCXX_USE_CXX11_ABI=1)”  but it didn’t help too.</p>
<p>After many tries I decided to built pytorch but then I got a new error for c10, the error log can be found here  <a href="https://gist.github.com/sanketgujar/8c55c8536910a0ede84133688fb5af41" rel="nofollow noopener">https://gist.github.com/sanketgujar/8c55c8536910a0ede84133688fb5af41</a>. I tried changes in the previous similar post but nothing is helping. The TORCH_CXX_FLAGS D_GLIBCXX_USE_CXX11_ABI=1 flag is already 1 for me as default in the TorchConfig.cmake.</p>
<p>I am using ROS Kinetic and my gcc version is 5.4</p>

username: yf225
cooked: <p>I think there are a few things we need to check:</p>
<ol>
<li>
<p>Is libtorch built with <code>_GLIBCXX_USE_CXX11_ABI=1</code>? (i.e. did you see <code>-- Determined _GLIBCXX_USE_CXX11_ABI=1</code> in the libtorch build log?)</p>
</li>
<li>
<p>Are you building <code>vehicle_detection</code> with the same compiler and with <code>_GLIBCXX_USE_CXX11_ABI=1</code>?</p>
</li>
</ol>

username: sanketgujar
cooked: <p>Hi Will,</p>
<p>Yes I made sure both are using the same ABI. I am using<br>
add_definitions(-D_GLIBCXX_USE_CXX11_ABI=1) for ROS package cmake files and changed the TorchConfig.cmake for the pytorch one.</p>
<p>Thanks</p>

username: sanketgujar
cooked: <p>Its been 9 days and I am still stuck, would someone please help me out</p>

username: yf225
cooked: <p>We now have new ABI binaries for libtorch. They can be found on<br>
<a href="http://pytorch.org/" rel="nofollow noopener">http://pytorch.org</a>, or at <a href="https://github.com/pytorch/pytorch/issues/17492#issuecomment-524692441" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/17492#issuecomment-524692441</a>. Can you try and see if it solves the problem?</p>
How to link libtorch in ros?

username: jinfagang
cooked: <p>Does anyone build robot with deep learning powered by pytorch??</p>
<p>I have a conflict situation now:</p>
<ul>
<li>libtorch built with gcc4.9, it using old ABI which can not link with higher programs.</li>
<li>ros does not support lower ABI</li>
</ul>
<p>To make libtorch link, I have to disable CXX11_ABI.<br>
but once I disable it, ros can not link to their libs.</p>
<p>if you don’t beleive , you can add this line your ros package, and your ros package can not build and link.</p>
<pre><code class="lang-auto">add_compile_options(-std=c++11 -D_GLIBCXX_USE_CXX11_ABI=0)
</code></pre>
<p>Any big god could help me out???</p>

username: ptrblck
cooked: <p>Hi Jin,</p>
<p>I think it would be a better idea to keep these topics together, as they are apparently all related to <code>libtorch</code> and <code>ROS</code>:<br>
<a href="https://discuss.pytorch.org/t/does-there-anybody-could-provide-an-example-project-integrate-ros-with-libtorch/48983">topic2</a><br>
<a href="https://discuss.pytorch.org/t/sos-libtorch-conflict-with-ros/48977">topic3</a></p>
<p>Otherwise different users might answer the same aspect multiple times (or try to debug it).</p>
<p>Would you mind collecting all these information in this topic?</p>

username: jinfagang
cooked: <p>ok, just hope someone can help me out</p>

username: yf225
cooked: <p>We now have new ABI binaries for libtorch. They can be found on<br>
<a href="http://pytorch.org/" rel="nofollow noopener">http://pytorch.org</a>, or at <a href="https://github.com/pytorch/pytorch/issues/17492#issuecomment-524692441" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/17492#issuecomment-524692441</a>.</p>
Why is pytorch 1.2 50% slower than pytorch 1.0?

username: achaiah
cooked: <p>I’m at a loss. I just upgraded from pytorch 1.0 to pytorch 1.2 and see huge slowdowns in training (50%-70%). What could be causing this discrepancy? I’m building my docker container off of official nvidia/cuda:10.0 image and haven’t changed anything except upgrading pytorch.</p>
<p>The command for upgrading was:</p>
<p><code>conda install -c pytorch pytorch=1.2.0=py3.7_cuda10.0.130_cudnn7.6.2_0 torchvision</code></p>
<p>What am I missing?</p>

username: ptrblck
cooked: <p>Do you have a code snippet so that we can reproduce this issue?</p>

username: achaiah
cooked: <p>Hmm… not easily, unfortunately. <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:">  I’m using a large custom dataset for segmentation with SGD and BCE + Dice so nothing too crazy there. The model I’m using is <a href="https://github.com/achaiah/pywick/blob/master/pywick/models/segmentation/gcnnets/gcn_resnext.py" rel="nofollow noopener">here</a>  I’ll try another model just to see if that’s where the issue lies.</p>

username: achaiah
cooked: <p>Confirming that that particular model is training much slower in pytorch 1.2.  Any ideas what specifically in it could be causing the slowdown?  At first glance there doesn’t seem to be anything incredibly different about it.</p>

username: ptrblck
cooked: <p>Could you try to profile specific parts of the model and try to isolate a submodule?<br>
I assume we could use some random dummy data for profiling, of the slowdown is created in the model?</p>

username: achaiah
cooked: <p>Ok I think I narrowed it down to the basic <a href="https://github.com/achaiah/pywick/blob/master/pywick/models/classification/resnext.py#L95" rel="nofollow noopener"><code>ResNeXt101_64x4d</code> class</a> since using a different backbone (e.g. densenet) does not produce the slowdown.  I’m guessing the slowdown is in the <a href="https://github.com/achaiah/pywick/blob/master/pywick/models/classification/resnext_features/resnext101_64x4d_features.py" rel="nofollow noopener">features</a>.  Still bewildered as to why that would cause such a drastic change in performance though.</p>