akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
My implementation of self attention

username: omer_sahban
cooked: <p>Hi everyone<br>
I’ve implemented 2 slightly different versions of multihead self-attention. In my head they should be equivalent to each other, but they’re giving different outputs even if all the weights and inputs are the exact same. where is the problem? which one is correct?<br>
v1 (modified from: <a href="https://github.com/pbloem/former/blob/master/former/modules.py" rel="nofollow noopener">https://github.com/pbloem/former/blob/master/former/modules.py</a>):</p>
<pre><code class="lang-auto">class MHSA(nn.Module):
  def __init__(self,
         emb_dim,
         kqv_dim,
         num_heads=1):
    super(MHSA, self).__init__()
    self.emb_dim = emb_dim # dimention of input&amp;output
    self.kqv_dim = kqv_dim  # dimention per head
    self.num_heads = num_heads # number of heads

    self.w_k = nn.Linear(emb_dim, kqv_dim * num_heads, bias=False)
    self.w_q = nn.Linear(emb_dim, kqv_dim * num_heads, bias=False)
    self.w_v = nn.Linear(emb_dim, kqv_dim * num_heads, bias=False)
    self.w_out = nn.Linear(kqv_dim * num_heads, emb_dim)

  def forward(self, x):
    b, t, _ = x.shape
    e = self.kqv_dim
    h = self.num_heads

    keys = self.w_k(x).view(b, t, h, e)
    values = self.w_v(x).view(b, t, h, e)
    queries = self.w_v(x).view(b, t, h, e)

    keys = keys.transpose(1, 2).contiguous().view(b * h, t, e)
    queries = queries.transpose(1, 2).contiguous().view(b * h, t, e)
    values = values.transpose(1, 2).contiguous().view(b * h, t, e)
    
    dot = torch.bmm(queries, keys.transpose(1, 2))
    dot = dot / np.sqrt(e)
    dot = F.softmax(dot, dim=2)
    out = torch.bmm(dot, values).view(b, h, t, e)
    out = out.transpose(1,2).contiguous().view(b, t, h * e)
    out = self.w_out(out)
</code></pre>
<p>v2:</p>
<pre><code class="lang-auto">class MHSA(nn.Module):
  def __init__(self,
         emb_dim,
         kqv_dim,
         num_heads=1):
    super(MHSA, self).__init__()
    self.emb_dim = emb_dim
    self.kqv_dim = kqv_dim
    self.num_heads = num_heads

    self.w_k = nn.Linear(emb_dim, kqv_dim * num_heads, bias=False)
    self.w_q = nn.Linear(emb_dim, kqv_dim * num_heads, bias=False)
    self.w_v = nn.Linear(emb_dim, kqv_dim * num_heads, bias=False)
    self.w_out = nn.Linear(kqv_dim * num_heads, emb_dim)

  def forward(self, x):

    b, t, _ = x.shape
    e = self.kqv_dim
    h = self.num_heads
    keys = self.w_k(x).view(b, t, h, e)
    values = self.w_v(x).view(b, t, h, e)
    queries = self.w_q(x).view(b, t, h, e)

    keys = keys.transpose(2, 1)
    queries = queries.transpose(2, 1)
    values = values.transpose(2, 1)

    dot = queries @ keys.transpose(3, 2)
    dot = dot / np.sqrt(e)
    dot = F.softmax(dot, dim=3)

    out = dot @ values
    out = out.transpose(1,2).contiguous().view(b, t, h * e)
    out = self.w_out(out)
    return out
</code></pre>

username: omer_sahban
cooked: <p>I can’t believe I made this silly mistake… in verson1 queries are outputted from w_v, instead of w_q.</p>
Different outcomes with nn.Sequential and nn.Functional

username: millanp95
cooked: <p>Hi, I have defined the following 2 architectures using some valuable suggestions in this forum. In my opinion they are the same, but I am getting very different performance after the same number of epochs.  The only difference is that one of them uses nn.Sequential and the other doesn’t. Any ideas?</p>
<p>The first architecture is the following:</p>
<pre><code class="lang-python">class Net(nn.Module):
    
    def __init__(self, n_input, n_output, w ):
        super(Net, self).__init__()
        self.features = nn.Sequential(
            nn.Conv2d(1, 32, w),
            nn.ReLU(),
            nn.Conv2d(32, 64, 3),
            nn.ReLU(),
            nn.MaxPool2d((2, 2)),
            nn.Dropout(p=0.25)
        )
        self.flat_fts = self.get_flat_fts(n_input, self.features)
        self.classifier = nn.Sequential(
            nn.Linear(self.flat_fts, 512),
            nn.ReLU(),
            nn.Linear(512, n_output), #Always check n_input here. 
            nn.LogSoftmax(dim=1)
        )
    
    def get_flat_fts(self, in_size, fts):
        f = fts(Variable(torch.ones(1, 1, in_size, in_size)))
        return int(np.prod(f.size()[1:]))
    
    def forward(self, x):
        fts = self.features(x)
        flat_fts = fts.view(-1, self.flat_fts)
        out = self.classifier(flat_fts)
        return out
</code></pre>
<p>The second one is the following:</p>
<pre><code class="lang-python">class Net(nn.Module):
    def __init__(self, n_input, n_output, w):
        super(Net,self).__init__()
        
        self.conv1 = nn.Conv2d(1, 32, w)
        self.conv2 = nn.Conv2d(32, 64, w)
        
        #Addition of the Linear layers
        self.fc1 = nn.Linear(64*30*30, 512) 
        self.fc2 = nn.Linear(512, n_output)

        #droput Layer. 
        self.droput = nn.Dropout(p=0.25)

        #output layer. 
        self.output = nn.Softmax(dim=1)
        
  
    def forward(self, x):
    
        x = F.relu(self.conv1((x)))
  
        # maxpoling over a (2,2) window, after using a RelU as activation function      
        x = F.max_pool2d(F.relu(self.conv2(x)), 2)
        x = self.droput(x)
        
        x = x.view(-1, self.num_flat_features(x))
        
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        
        return self.output(x)
    
    def num_flat_features(self, x):
        size = x.size()[1:]  # all dimensions except the batch dimension
        num_features = 1
        for s in size:
            num_features *= s
        return num_features

</code></pre>

username: albanD
cooked: <p>One has a Softmax for output and one has a LogSoftmax. Is that expected?</p>

username: millanp95
cooked: <p>No, that might be it. Thank you.</p>
Efficient Computation of Distance Metrics Between Different Sized Tensors

username: kheyer
cooked: <p>Here’s the basic idea. I want to compute some distance metric (Euclidian, cosine, etc) between a vector and a matrix of vectors. In the case of a single vector, this is fairly straightforward.</p>
<pre><code class="lang-auto">a1 = torch.randn(1,5)
a2 = torch.randn(12,5)
distances = distance_function(a1.expand_as(a2),  a2) # returns vector of size 12
</code></pre>
<p>However things get tricky in the batch case where the sizes of the second matrix can vary. For example:</p>
<pre><code class="lang-auto">a1 = torch.randn(1,5)
b1 = torch.randn(1,5)
c1 = torch.randn(1,5)

a2 = torch.randn(12,5)
b2 = torch.randn(7,5)
c2 = torch.randn(2,5)
</code></pre>
<p>I’d like to compute all distance comparisons (a1 to a2, b1 to b2, c1 to c2, etc) in a way that is fast and memory efficient. Memory efficiency is important because sometimes the x_2 matrices can be large.</p>
<p>The first approach that comes to mind is something like this:</p>
<pre><code class="lang-auto">distances = distance_function(torch.cat([a1.expand_as(a2), b1.expand_as(b2), c1.expand_as(c2)]),
                    torch.cat([a2,b2,c2]))
</code></pre>
<p>This works but I have concerns about the memory implications. I know <code>expand_as</code> does not allocate new memory. However (to my knowledge), using <code>torch.cat</code> will result in new memory allocation.</p>
<p>Is there a better/more efficient approach to this?</p>

username: albanD
cooked: <p>If you don’t have too many of these Tensors, I think a for loop around your original distance function above is the best <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>If you have many many of them, then the cat operation will be faster (though more memory hungry as you mentionned).</p>
Should I normalize with ImageNet stats if finetuning network?

username: xian_kgx
cooked: <p>Hi all, just to ask on your opinions if I should be normalizing images with ImageNet stats if I’m finetuning a network.</p>
<p>I understand I should use the same preprocessing if I’m to use a pretrained network as is without modification, but what if I’m finetuning?</p>
<p>I think it shouldn’t matter, but please let me know of your thoughts. Thanks.</p>

username: dejanbatanjac
cooked: <p>What is finetuning?</p>
<p>You should use the normalization technique that has been used for the training. Since ImageNet pretrained models used the ImageNet stats, you should use that if you plan to use a model that has been trained on ImageNet.</p>
<p>If finetuning means you alter the number of classes for the classification model, or you are using some semantic segmentation on an ImageNet pretrained backbone, then you should still use this ImageNet stats.</p>

username: xian_kgx
cooked: <p>Let’s take your first example of altering the number of classes. Since the weights on the head of the network is not trained (initialized randomly), shouldn’t gradient descent be smart to modify the parameters in those new layers to adapt accordingly.</p>

username: dejanbatanjac
cooked: <p>Do you use any ImageNet pretrained network? If answer is Yes, then you should normalize your input images with ImageNet stats.</p>
<p>If answer is No, then you should not use ImageNet stats, instead you should use the stats based on your training images.</p>
<p>Let’s say you have 10.000 training images and 1.000 validation images.<br>
You may calculate the mean and standard deviation for your 10.000 images for all 3 channels if you have 3 channels and you should use that for both the training and validation set to normalize the images.</p>
<p>It won’t be wrong to use all 11.000 images and to calculate the mean and std and use that.<br>
In fact it would be even better.</p>
<p>But once you get these numbers you need to stick with them.</p>

username: xian_kgx
cooked: <p>Thanks for your suggestion.</p>

username: amit_js
cooked: <p><a class="mention" href="/u/dejanbatanjac">@dejanbatanjac</a>, Please help me understand this.<br>
<strong>"It won’t be wrong to use all 11.000 images and to calculate the mean and std and use that.</strong><br>
<strong>In fact it would be even better."</strong><br>
Till now I have seen people only using the mean and std of train set only. How will doing that for the entire dataset work?<br>
Thanks in advance.</p>
Are Modules signaled updates by the Optimizer

username: marioviti
cooked: <p>Hi to all!</p>
<p>As fare as I understand it, an object of the th.nn.optim.Optimizer class updates parameters of a torch.nn.Module object by reference, so the Module is unaware of the updates.<br>
Is there a way to put a monitor on the weight.data and bias.data? Is a Module signalled somehow during updates of its parameters?</p>
<p>Thank you for your help!</p>

username: albanD
cooked: <p>Hi,</p>
<p>You should never use <code>.data</code> as it is deprecated now.</p>
<p>I don’t think there is any mechanism to know when a Tensor is modified I’m afraid.  The Optimizer does not even reference the Module, just the Tensors.<br>
What would be your particular use case here?</p>

username: marioviti
cooked: <p>Hi!<br>
Thanks for your reply!<br>
I’m trying to share updates among variables processing different features.<br>
After some try-errors I just found out that it’s just the same as copying the individual variables as all updates will be the sames.<br>
But now I’m concerned… I’m using <code>.data</code> to copy the same variable!!!</p>
CUDA out of memory - VGG16

username: dersu
cooked: <p>I have been looking for an answer on how to load a VGG16 model on a 12 GB GPU and not getting a CUDA out of memory error, but I could not find it yet. The sizes may differ with different architectures of VGG, but I get this error message:</p>
<blockquote>
<p>RuntimeError: CUDA out of memory. Tried to allocate 5.49 GiB (GPU 0; 10.92 GiB total capacity; 6.27 GiB already allocated; 4.09 GiB free; 20.64 MiB cached)</p>
</blockquote>
<p>I have tried parallelizing the model by increasing the GPU count, but I think we are not able to do that. Is there any way to implement a VGG16 model with 12 GB GPUs? Any help would be appreciated. Thanks.</p>

username: luodahei
cooked: <p>you could reduce the input size?</p>

username: gost-sniper
cooked: <p>You can solve that problem by reducing the batch size or/and image dimensions</p>

username: RicLaGra
cooked: <p>I had the same problem. The main reason is that you try to load all your data into gpu.<br>
A possible solution is to reduce the batch size and load into gpu only few data per time and finally after your computation to send from gpu to cpu your data .cpu(). (but it depends… i don’t know your code).<br>
However, you should pay attention if your GPU is free (because is possible it is busy by another process). Let me know.</p>

username: Erdi
cooked: <p>I had the same problem before. After every batch in the training and testing adding</p>
<p>torch.cuda.empty_cache()</p>
<p>worked for me. However it is always better to watch the gpu stats. You can use <a href="https://pypi.org/project/gpustat/" rel="nofollow noopener">gpustat</a>. For example if you run the code and dont restart the kernel for the next run, some data from the previous run stuck in GPU memory.</p>

username: ansuman_das
cooked: <p>First you have to calculate it’s model space(sum of input size, forward and backward size, params size ) The you have to change the input size or batch size to see if your model space fits your gpu or not(&lt;12 gb for your case). If it fits then it’s good to go. You can use torchsummary package to calculate the model size.</p>
Can I specify backward() function in my custom layer by inheriting nn.Module?

username: Megh_Bhalerao
cooked: <p>Hi,<br>
I am trying to implement a standard gradient reversal layer which looks something like this:</p>
<pre><code class="lang-auto">class GradientReversalModule(nn.Module):
    def __init__(self,lambd):
        super(GradientReversalModule,self).__init__()
        self.lambd = lambd
    def forward(self,x):
        return x
    def backward(self,grad_value):
        return -grad_value*self.lambd
</code></pre>
<p>I am just confused if I am supposed to inherit from nn.Module or torch.autograd.Function? And also, what is the difference between the two? Please do let me know, thanks!</p>

username: albanD
cooked: <p>Hi,</p>
<p>An nn.Module is just a convenient construct to handle parameters, buffers, interaction with optimizers in the context of torch.nn</p>
<p>A autograd.Function is a new elementary op in the autograd.</p>
<p>So if you want to specify the backward for a given op, you want a custom autograd.Function. See the doc <a href="https://pytorch.org/docs/stable/notes/extending.html">here</a> on how to do that.</p>

username: Megh_Bhalerao
cooked: <p>Thanks for letting me know <a class="mention" href="/u/alband">@albanD</a>!!</p>
BatchNorm2d when batch size 1 works, what is it doing?

username: mikolchon
cooked: <p><code>BatchNorm2d</code> works even when batch size is 1, which puzzles me. So what is it doing when batch size is 1? The only related thread I could find is <a href="https://github.com/pytorch/pytorch/issues/1381" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/1381</a> without much explanation.</p>
<p>minimal example:</p>
<pre><code class="lang-auto">x = torch.FloatTensor(1,1,2,2)
x[0,0,0,:] = 1
x[0,0,1,:] = 2
m = nn.BatchNorm2d(1)
y = m(Variable(x))

// output is
// (0,0,.,.) =  [[-0.1544, -0.1544], [0.1544, 0.1544]]
</code></pre>

username: SimonW
cooked: <p>There are the gamma and betas. They are initialized as gamma ~ U[0, 1], and beta &lt;- 0. if you do</p>
<pre><code class="lang-python">x = torch.FloatTensor(1,1,2,2)
x[0,0,0,:] = 1
x[0,0,1,:] = 2
m = nn.BatchNorm2d(1)
m.weight.data.fill_(1)
m.bias.data.zero_()
y = m(Variable(x))
# you will get
&gt;&gt;&gt; y

(0 ,0 ,.,.) =
 -1.0000 -1.0000
  1.0000  1.0000
[torch.FloatTensor of size (1,1,2,2)]

</code></pre>

username: dio_din
cooked: <p>After <code>y = m(x)</code>，<code>m.running_mean = 0.1500</code> and <code>m.running_var = 0.9333</code>.<br>
Why the resulting <code>y</code> is not <code>(x - 0.15) / sqrt(0.9333)</code>, which is</p>
<pre><code class="lang-auto">tensor([[[[0.8798, 0.8798],
          [1.9149, 1.9149]]]])
</code></pre>
<p>?</p>
Average the matrix along the lines

username: slavavs
cooked: <p>I have a I have a matrix [5.5].<br>
I have to average and get the matrix [1,5].<br>
How do I do that? [5,5].<br>
I have to average and get the matrix [1,5].<br>
How do I do that?</p>

username: albanD
cooked: <p>Hi,</p>
<p>You can specify the dimension you want to reduce over in the mean functions:<br>
<code>A.mean(dim=0)</code>.</p>
LINK : fatal error LNK1104: cannot open file 'torch.cpu'

username: justanhduc
cooked: <p>Hello. I got this problem when just include <code>torch/torch.h</code>. My environment is visual studio 2019, MSVC v142 build tools v14.25, libtorch 1.5 CPU. Thanks!<br>
EDIT1: I use the debug version with debug configuration.</p>

username: peterjc123
cooked: <p>What is <code>torch.cpu</code>? Do you mean <code>torch_cpu.lib</code>?</p>

username: justanhduc
cooked: <p>Thanks for your response.<br>
No, it is <code>torch.cpu</code>. I copied the error as it is.<br>
The program runs fine with the release version/release configuration, but I can’t get debug/debug to work.</p>

username: peterjc123
cooked: <p>How could a file get a name like <code>torch.cpu</code>? How did you create your project?</p>

username: justanhduc
cooked: <p>Sorry I just checked the input of linker again. I made a stupid typo by including that file.</p>
Hessian with ReLUs?

username: Sam_Lerman
cooked: <p>I’m using the new autograd functional API and I have a question about deriving a ReLU-activated network. Is it possible to compute a hessian of the output of a neural network with respect to the inputs using ReLU activations? I imagine it would be if the output was activated by a sigmoid or softmax, but what if the output is just raw?</p>

username: albanD
cooked: <p>Hi,</p>
<p>If the output is “raw” then your network as a whole is piece-wise linear.<br>
So the Hessian will just be full of 0s <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> (except if you evaluate at a point where it is non-differentiable).</p>
Memory_allocated() vs memory_cached()

username: nionjo
cooked: <p>Hello,</p>
<p>From PyTorch documentation, remains unclear to me which is the difference between the two commands: it seems to me that cached memory is a superset of allocated memory. Could you give me a more detailed explanation on this?</p>

username: albanD
cooked: <p>Hi,</p>
<p>The allocated memory is the memory that is currently used to store Tensors on the GPU.<br>
The cached memory is the memory that is currently used on the GPU by pytorch (as can be seen in nvidia-smi).</p>

username: Wong_Jones
cooked: <p>Hi albanD, I am still confused. According to your explanation, cached should include allocated (as the Tensors are used by pytorch). However, I encountered the error “Tried to allocate 5.57 GiB (GPU 0; 15.77 GiB total capacity; 5.83 GiB already allocated; 3.29 GiB free; 5.51 GiB cached)”. It seems that the allocated is larger than cached. Besides, Even though there is only my process running on “cuda:0”, the “GPU memory usage” is always inconsistent with cached. Could you give me a help on how to estimate the gpu memory could be used further? I am currently using free + (cached - allocated).</p>

username: albanD
cooked: <p>Hi,</p>
<p>This is quite an old comment.<br>
You can find up to date informations on the doc <a href="https://pytorch.org/docs/stable/notes/cuda.html#cuda-memory-management">here</a>.</p>
Can't download IMDB datasets in torchtext

username: Eric1012
cooked: <p>Hi, it seems that I didn’t download IMDB datasets from torchtext.datasets and the train_data is empty. It’s so strange. I wonder how to fix it.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/4/d43cce3711283585930f8266a1702ad6fb7c0858.png" data-download-href="https://discuss.pytorch.org/uploads/default/d43cce3711283585930f8266a1702ad6fb7c0858" title="2020-05-13 (3)"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/4/d43cce3711283585930f8266a1702ad6fb7c0858_2_690x370.png" alt="2020-05-13 (3)" data-base62-sha1="uhxuW4kzSO0zL2rGVvvv5BE1Ug0" width="690" height="370" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/4/d43cce3711283585930f8266a1702ad6fb7c0858_2_690x370.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/4/d43cce3711283585930f8266a1702ad6fb7c0858_2_1035x555.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/4/d43cce3711283585930f8266a1702ad6fb7c0858_2_1380x740.png 2x" data-dominant-color="F5F6F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">2020-05-13 (3)</span><span class="informations">1419×761 25.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: albanD
cooked: <p>Hi, could it be that the token you ask are the issue?<br>
Is the approach given in the doc <a href="https://torchtext.readthedocs.io/en/latest/datasets.html">here</a> works?</p>

username: Eric1012
cooked: <p>The approach1 can work. But it seems I didn’t download the IMDB datasets successfully.</p>
InternalError: Blas GEMM launch failed : a.shape=(128, 122), b.shape=(122, 64), m=128, n=64, k=122

username: arezoo_moradi
cooked: <p>When I run this model using python:</p>
<pre><code> def build_network():

models = []
model = Sequential()
model.add(Dense(64, input_dim=122))

model.add(Dense(32))
model.add(Activation('relu'))
model.add(Dropout(.15))
model.add(Dense(32))
model.add(Activation('relu'))
model.add(Dropout(.15))
model.add(Dense(32))
model.add(Activation('relu'))
model.add(Dropout(.15))
model.add(Dense(5))
model.add(Activation('softmax'))

model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

return model
</code></pre>
<p>NN = build_network()<br>
early_stopping = EarlyStopping(monitor=‘val_loss’, min_delta=0, patience=3, verbose=0, mode=‘auto’)</p>
<p>NN.fit(x=train, y=target, epochs=100, validation_split=0.1, batch_size=128, callbacks=[early_stopping])</p>
<p>I receive this error:</p>
<p>InternalError: Blas GEMM launch failed : a.shape=(128, 122),<br>
b.shape=(122, 64), m=128, n=64, k=122    [[{{node dense_6/MatMul}}]]</p>

username: Nikronic
cooked: <p>Hi,</p>
<p>This is Keras/TF code, not PyTorch.</p>
Prediction output

username: Abhishek_Kumar3
cooked: <p>Hello. While predicting probability using sigmoid activation output will be in the form of<br>
Prediction.                                   Label<br>
Tensor([0.95,0.123])               tensor([1.,0.])<br>
Tensor([0.96,0.42])                 tensor([0.,1.])<br>
…so on<br>
Can u please tell me why that 2 probability values are displayed?. And in Label why once it well display 1,0 fit some and 0,1 for some images?</p>

username: futscdav
cooked: <p>Because you have 2 classes.</p>

username: Abhishek_Kumar3
cooked: <p>Yes. Now how can I classify using these values?<br>
Tell me how can I classify images that belongs to class 0 and also class 1?</p>

username: futscdav
cooked: <p>Typically you would classify with whatever probability is highest.</p>
<pre><code class="lang-auto">Tensor([0.95,0.123]) tensor([1.,0.])
Tensor([0.96,0.42]) tensor([0.,1.])
</code></pre>
<p>Would mean both examples are classified as class 0. Second one has ground truth label of class 1.</p>

username: Abhishek_Kumar3
cooked: <p>I didn’t understood Sir.  How it is classified as class 0. Because in second case probability value is more for label 0. In first case it is opposite .</p>
Error: OpenCV(3.4.2) error: (-209:Sizes of input arguments do not match) The operation is neither 'array op array' (where arrays have the same size and the same number of channels), nor 'array op scalar', nor 'scalar op array' in function 'cv::arithm_op'

username: Omar2017
cooked: <p>Hi everyone<br>
I am currently working on keyframe extraction from videos.</p>
<p>Code :</p>
<pre><code class="lang-auto">import torch
from torch.utils.data import Dataset
from PIL import Image
import numpy as np
import cv2

class makeDataset(Dataset):
    def __init__(self, dataset, labels, spatial_transform, seqLen=20):
        self.spatial_transform = spatial_transform
        self.images = dataset
        self.labels = labels
        self.seqLen = seqLen

    def __len__(self):
        return len(self.images)

    def __getitem__(self, idx):
        vid_name = self.images[idx]
        label = self.labels[idx]
        inpSeq = []
        self.spatial_transform.randomize_parameters()
        vid=cv2.VideoCapture(vid_name)
        ret, prev_frame = vid.read()
        p_frame_thresh = 20
        count=1
        index=0
        while(index&lt;=self.seqLen):
            # Extract images
            ret, curr_frame = vid.read()
            diff = cv2.absdiff(curr_frame, prev_frame)
            non_zero_count = np.count_nonzero(diff)
            if non_zero_count &gt; p_frame_thresh:
                count+=1
                img=Image.fromarray(curr_frame)
                inpSeq.append(self.spatial_transform(img.convert('RGB')))
            prev_frame = curr_frame
        print (count)
        inpSeq = torch.stack(inpSeq, 0)
        return inpSeq, label
</code></pre>
<p>what is the problem here, I don’t understand this error? can any one guide me please <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: Abhijith_M
cooked: <p>The two frames in video not having the same size.</p>
<p>diff = cv2.absdiff(curr_frame, prev_frame)</p>
<p>this operation only performs when the two frames having same size.</p>

username: Sachin_Jadhav
cooked: <p>I am also having the same problem… How do you fix it ?  I am new to Python and learning now… I hope that there is a way to make the images of same sizes using code…</p>
Test acc. slightly higher than train acc for large number of epochs

username: erm
cooked: <p>I have a simple fully connected NN that I tested with:<br>
CIFAR10,<br>
FashionMNIST,<br>
MNIST and<br>
Cat vs. non Cat datasets.<br>
For all the 4 datasets I observe normal behavior in the accuracy, namely, train accuracy is higher than test acc. But, for two cases of MNIST, I get about 1 % higher accuracy for test than for train (90% train acc. and 91%test acc.). This was when running 500 epochs.<br>
I thought to reduce the number of epochs to 300 and see what happened, and the results were then: 90% train acc. and 89% test acc.<br>
How to interpret this?</p>

username: futscdav
cooked: <p>One possible explanation is that the model doesn’t have enough capacity to overfit. See what happens when you make the model larger.</p>

username: erm
cooked: <p>So, how would one explain that the same architecture does not face the problem with CIFAR10 that is more difficult to train? i will try your suggestion and see what happens.</p>

username: futscdav
cooked: <p>Depending on the accuracy achieved, CIFAR10 might not be necessarily a harder dataset. E.g. classifying by prevalent color will get you to some accuracy on CIFAR, and you will almost certainly overfit on training data.</p>
<p>Hard to say without knowing about the results and overall setup.</p>
Shared CUDA memory with torch.distributed

username: LouisKirsch
cooked: <p>I’m using torch.distributed with MPI and would like to share a CUDA tensor across my MPI world.<br>
From <a href="https://pytorch.org/docs/stable/multiprocessing.html#sharing-cuda-tensors" rel="nofollow noopener">https://pytorch.org/docs/stable/multiprocessing.html#sharing-cuda-tensors</a> it looks like shared CUDA memory is possible with torch.multiprocessing<br>
How can I do the same with torch.distributed?</p>
<p>Thanks!</p>
Tensor dimensionality and speed impact

username: Philipp_Hager
cooked: <p>Newbie question here. In my research I found <a href="https://www.quora.com/How-do-I-determine-the-number-of-dimensions-for-word-embedding" rel="noopener nofollow ugc">this thread</a> on word embedding dimensions stating:</p>
<blockquote>
<p>You can pick a power of 32 (64, 128, 256) to speed up modeling training depending on your software framework and hardware.</p>
</blockquote>
<p>Is that also a property in PyTorch and is using a power of 32 increasing speed?</p>

username: futscdav
cooked: <p>Generally speaking, multiples of 8 or 32 will perform better on most GPUs. Depends on your specific setup, but it’s not something I’d prioritize too much unless you are just optimizing.</p>
Module 'torchvision.models' has no attribute 'video'

username: Subhankar_Ghosh
cooked: <p>I am using PyTorch 1.5 on Ubuntu 16.04 installed via anaconda, cuda 10.1<br>
My model:</p>
<pre><code class="lang-auto">    model = models.video.r3d_18(pretrained=True, progress=False)
    num_features = model.fc.in_features
    model.fc = nn.Linear(num_features, num_classes)
</code></pre>
<p>I am getting this error:</p>
<blockquote>
<p>AttributeError: module ‘torchvision.models’ has no attribute ‘video’</p>
</blockquote>
<p>Can you all please help me out? Thanks!</p>

username: futscdav
cooked: <p>Make sure the installed version of torchvision is &gt;= 0.4.0.</p>
<p>Try</p>
<pre><code class="lang-auto">import torchvision
print(torchvision.__version__)
</code></pre>
<p>if you are unsure what version is installed.</p>

username: Subhankar_Ghosh
cooked: <aside class="quote no-group" data-username="futscdav" data-post="2" data-topic="81120">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/f/a183cd/48.png" class="avatar"> futscdav:</div>
<blockquote>
<pre><code class="lang-auto">import torchvision
print(torchvision.__version__)
</code></pre>
</blockquote>
</aside>
<p>I reinstalled pytorch by creating an environment using pip and:</p>
<p><code>pip install torch==1.5.0+cu101 torchvision==0.6.0+cu101 -f https://download.pytorch.org/whl/torch_stable.html</code></p>
<p>But after trying:</p>
<pre><code class="lang-auto">import torchvision
print(torchvision.__version__)
</code></pre>
<p>the output is:</p>
<blockquote>
<p>0.2.2</p>
</blockquote>
<p>This is strange.</p>

username: Subhankar_Ghosh
cooked: <p>I created a new environment with anaconda again and I am getting the same.</p>

username: futscdav
cooked: <p>Try <code>pip uninstall torchvision</code> until there are no installations left. Then install it again with <code>pip install torchvision</code>.</p>
Select a detection tensor output where its confidence above threshold

username: Kiki_Rizki_Arpiandi
cooked: <p>I try to achieve this code</p>
<p><code>pred = pred[pred[:, 4] &gt; opt.conf_thres]</code></p>
<p>Pred has [Nx518] where n is the number of detection</p>
<pre><code class="lang-auto">auto pred = detection_model.forward_model(im_blob).at(0).squeeze();
//pred has shape [Nx518]
auto conf_mask = (pred.select(1, 4) &gt; opt.conf_thres);
auto filtered_index = conf_mask.nonzero().squeeze();
pred = torch::index_select(pred, 0, filtered_index);
</code></pre>
<p>Is my C++ solution considered bad practice ? is there any other solution?</p>

username: yuangan.zhou
cooked: <p>I have the same question with you too, do you get the answer?</p>
Using the gradients in a loss function

username: stefdoerr
cooked: <p>Hi.<br>
Say I know both what the output of my network should be and what the gradients ought to be for each training example.<br>
Is it possible to train a network in pytorch in which the loss depends on both the output of the network and the gradients of the network w.r.t. the network input?</p>
<pre><code class="lang-python">inp = Variable(torch.rand(3, 4), requires_grad=True)
W = Variable(torch.rand(4, 4), requires_grad=True)
yreal = Variable(torch.rand(3, 4), requires_grad=False)
gradsreal = Variable(torch.rand(3, 4), requires_grad=True)

ypred = torch.matmul(inp, W)
ypred.backward(torch.ones(ypred.shape))
gradspred = inp.grad

loss = torch.mean((yreal - ypred) ** 2 + (gradspred - gradsreal) ** 2)
loss.backward()
</code></pre>
<p>This won’t work as <code>inp.grad</code> is volatile. Also would I have to zero all gradients after calculating the gradspred?</p>

username: SimonW
cooked: <p>To make it working w/o much change, add <code>create_graph=True, retain_graph=True</code> to the backward call.</p>
<p>However, a more efficient way is:</p>
<pre><code class="lang-auto">gradspred, = autograd.grad(ypred, inp, 
                           grad_outputs=ypred.data.new(y_pred.shape).fill_(1),
                           create_graph=True)
loss = ...
loss.backward()
</code></pre>

username: stefdoerr
cooked: <p>Great! Worked like a charm <img src="//discuss.pytorch.org/images/emoji/apple/smile.png?v=5" title=":smile:" class="emoji" alt=":smile:"></p>

username: 111239
cooked: <p>Hello! Thank you for your answer. I used your instruction and did not understand what does code “grad_outputs=ypred.data.new(y_pred.shape).fill_(1)” do. Can you please explain it or point me to explanation?</p>

username: nima_rafiee
cooked: <p>take look at <a href="https://discuss.pytorch.org/t/assign-manual-assigned-grad-output/2750" class="inline-onebox">Assign manual assigned "grad_output"</a></p>
How to estimate available gpu memory

username: Wong_Jones
cooked: <p>I am using pytorch 1.2 and use the following code to estimate the available gpu memory:<br>
h = nvmlDeviceGetHandleByIndex(0)<br>
info = nvmlDeviceGetMemoryInfo(h)<br>
<span class="hashtag">#print</span>(f’total    : {info.total}’)<br>
print(f’free     : {info.free}’)<br>
<span class="hashtag">#print</span>(f’used     : {info.used}’)<br>
print(‘torch cached: {}’.format(torch.cuda.memory_cached(0)))<br>
print(‘torch allocated: {}’.format(torch.cuda.memory_allocated(0)))<br>
remaining = info.free + (torch.cuda.memory_cached(0) - torch.cuda.memory_allocated(0))</p>
<p>in summary, I use free + (cached - allocated) as the available gpu memory for further allocating tensors</p>
<p>However, I encountered an error:<br>
RuntimeError: CUDA out of memory. Tried to allocate 5.57 GiB (GPU 0; 15.77 GiB total capacity; 5.82 GiB already allocated; 3.31 GiB free; 5.50 GiB cached)</p>
<p>I checked the code here: <a href="https://github.com/pytorch/pytorch/blob/54a63e0420c2810930d848842142a07071204c06/c10/cuda/CUDACachingAllocator.cpp#L244" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/54a63e0420c2810930d848842142a07071204c06/c10/cuda/CUDACachingAllocator.cpp#L244</a><br>
It seems that “cached” as the error info pointed is <code>stats.amount_cached - stats.amount_allocated</code><br>
In this sense, this part can be used and free (3.31G) + cached (5.50G) is larger than what I am going to allocate (5.57G).</p>
<p>So what’s the reason? I guess I misunderstand some mechanisms. Could you give me a help? Thanks!</p>

username: Wong_Jones
cooked: <p>actually in the runtime, I logged the cached and allocated. it seems that their difference is enough for the tensor to be allocated. So why does the error occur?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/5/75ac1a4b2878f1f0973a379aa3ddf2ecdb8fcba2.png" data-download-href="https://discuss.pytorch.org/uploads/default/75ac1a4b2878f1f0973a379aa3ddf2ecdb8fcba2" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/5/75ac1a4b2878f1f0973a379aa3ddf2ecdb8fcba2_2_690x353.png" alt="image" data-base62-sha1="gMYAJaIJLxhwTOeXpTjmIpZRdTk" width="690" height="353" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/5/75ac1a4b2878f1f0973a379aa3ddf2ecdb8fcba2_2_690x353.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/5/75ac1a4b2878f1f0973a379aa3ddf2ecdb8fcba2_2_1035x529.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/5/75ac1a4b2878f1f0973a379aa3ddf2ecdb8fcba2_2_1380x706.png 2x" data-dominant-color="111411"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1691×867 88.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
nn.ModuleList() seems not to support * operation?

username: feiyuhuahuo
cooked: <p>I found that if I construct one ModuleList by this way:</p>
<pre><code class="lang-auto">self.pre_layer = nn.ModuleList([nn.Sequential(nn.Conv2d(256, 256, kernel_size=3, padding=1),
                                                  nn.ReLU(inplace=True))] * 3)
</code></pre>
<p>This weights of the three Conv2d are always keeping the same.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/a/5/a58649da14e9d4ede08164609b631659aae709c0.png" alt="image" data-base62-sha1="nCiuEn9NIzyVCFs0hmBr3xl1WqA" width="523" height="109"><br>
But their ids are different:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/0/1/017b122b6688171e866145329e1de7542ffc6d25.png" alt="image" data-base62-sha1="d69EXJwwZsOWNBQVMe9wIG27Ln" width="499" height="106"><br>
What’s the reason?<br>
Besides, if use for loop to construct a ModuleList, then everything works well.</p>
<pre><code class="lang-auto">self.pre_layer = nn.ModuleList([nn.Sequential(nn.Conv2d(256, 256, kernel_size=3, padding=1),
                                                  nn.ReLU(inplace=True)) for i in range(3)])
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/c/dc66e71b37fb8e30af27825514535128ce75bad6.png" alt="image" data-base62-sha1="vrLvIKVBwBb6y4abNxB9vo9dyVo" width="521" height="109"></p>
FasterRCNN on COCO with different combination of Resnet50 backbones

username: Westerby
cooked: <p>Hello,<br>
I get very different results using apparently almost the same backbone for training COCO. As a basis I use code from <a href="https://github.com/pytorch/vision/blob/master/references/detection/train.py" rel="nofollow noopener">here</a>. Except for the network architecture all training parameters stay the same.</p>
<ol>
<li>
<p>Unmodified <code>maskrcnn_resnet50_fpn</code> model - I get:<br>
<code>Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.354</code><br>
and thats consistent with the results reported at Vision’s detection <a href="https://pytorch.org/docs/stable/torchvision/models.html#object-detection-instance-segmentation-and-person-keypoint-detection" rel="nofollow noopener">website</a></p>
</li>
<li>
<p>I get pretrained Resnet50 backbone and put it in FasterRCNN wrapper:</p>
</li>
</ol>
<pre><code class="lang-auto">bb_model = torchvision.models.resnet50(pretrained=True)
backbone = nn.Sequential(bb_model.conv1,
                      bb_model.bn1,
                      bb_model.relu,
                      bb_model.maxpool,
                      bb_model.layer1,
                      bb_model.layer2,
                      bb_model.layer3,
                      bb_model.layer4)
backbone.out_channels = 2048
model = FasterRCNN(backbone,num_classes=num_classes)
</code></pre>
<p>This model has no frozen Batchnorm and first two layers are not frozen either - plus there’s no FPN . But the result I get is strikingly different, something I didn’t expect:<br>
<code>Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.239</code></p>
<ol start="3">
<li>Third version I get the same backbone as in original training code, the difference is the final model has no FPN:</li>
</ol>
<pre><code class="lang-auto">resnet50_fpn = torchvision.models.detection.maskrcnn_resnet50_fpn(num_classes=91)
backbone_fpn = nn.Sequential(
	resnet50_fpn.backbone.body.conv1,
	resnet50_fpn.backbone.body.bn1,
	resnet50_fpn.backbone.body.relu,
	resnet50_fpn.backbone.body.maxpool,
	resnet50_fpn.backbone.body.layer1,
	resnet50_fpn.backbone.body.layer2,
	resnet50_fpn.backbone.body.layer3,
	resnet50_fpn.backbone.body.layer4
	)
backbone_fpn.out_channels = 2048
model = FasterRCNN(backbone_fpn,num_classes=num_classes)
</code></pre>
<p>The result with the same training protocol is a little bit better, but still something I didn’t expect:<br>
<code>Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.265</code></p>
<p>So the question is - why is that? Is that something wrong with combining backbone with FasterRCNN in this manner?</p>
Cosine similarity on 3D tensors and Filtering

username: srikara
cooked: <p>Hi All,<br>
I have two 3D tensors X and Q of shape (5, 16, 128) on which I do cosine similarity on 2nd dim to get a (5, 16) cosine-similarity vector. I then sort this cosine-similarity vector, to get indices of most-to-least similar vectors in the original vector Q. In code</p>
<pre><code class="lang-auto">X = torch.ones((5, 16, 128))
Q = torch.randn(5, 16, 128) # dim=2 is features
neg_samples = 11
pos_samples = 5

cos_sim = torch.nn.CosineSimilarity(dim=2, eps=1e-6)
sim_scores = cos_sim(X, Q) # sim_scores output shape: [5,16]
_, sorted_idxs = torch.topk(sim_scores, k=sim_scores.shape[1], sorted=True, dim=1) #sort sim_scores to get sorted indexes same shape [5, 16]

#since sorted_idxs is sorted descending on similarity, we can slice to divide pos/neg samples.
pos_idxs = sorted_idxs[:, :pos_samples]  #gives [5, 5] most positive idxs from sorted_idxs
neg_idxs = sorted_idxs[:, pos_samples:(pos_samples+neg_samples)] #gives [5, 11] most negative idxs from sorted_idxs
</code></pre>
<p>How do i retrieve the corresponding tensors from <code>Q</code> which is of shape <code>[5, 16, 128]</code> using these <code>pos_idxs</code> shape <code>[5,5]</code> and <code>neg_idxs</code> shape <code>[5,11]</code>?</p>
<p>I tried <code>Q[ pos_idxs, :]</code> but i get output tensor of shape <code>[5, 5, 16, 128]</code>. I want to slice <code>Q</code> into <code>[5, 5, 128]</code> and <code>[5, 11, 128]</code> two separate tensors based on 2D <code>pos_idxs</code> and  <code>neg_idxs</code> index tensors.</p>
<p>How do i do this in pytorch using tensor operations?</p>
<p>thanks</p>
Specify a sampler in BucketIterator

username: abhigenie92
cooked: <p>Getting a weighted sampler:</p>
<pre><code class="lang-auto">df = pd.read_csv(file_path)
class_sample_count_y = len(df[df["label"] == "Y"].index)
class_sample_count_n = len(df[df["label"] == "N"].index)
weights = []
for x in df["label"].tolist():
    if x == "Y":
        weights += [1./ class_sample_count_y]
    else:
        weights += [1./class_sample_count_n]
num_samples = len(weights)
weights = torch.Tensor(weights)
weights = weights.double()
train_sampler = torch.utils.data.sampler.WeightedRandomSampler(
    weights, num_samples)
</code></pre>
<p>How can I specify this for a buckeriterator?</p>
<pre><code class="lang-auto">train_iterator= data.BucketIterator.splits(
    train_data
    batch_size=BATCH_SIZE,
    sort_key=lambda x: len(x.raw_text),
    device=device,
)
</code></pre>
<p>There is no option for passing a sampler here.</p>
Validation Losses with Dropout/BN

username: nmtp
cooked: <p>I’m running some training on a large CNN with dropout and batchnorm layers and getting some funky validation loss numbers. I’ve read here that this is common when there are dropout layers in the NN. This might be a silly question but will the validation/training loss curves eventually converge? Or is there something else I have to do to accommodate for the large discrepancy between the values?</p>

username: ptrblck
cooked: <p>Dropout might cause your training loss to be higher than your validation loss.<br>
You could use <code>model.eval()</code> after training a complete epoch and calculate the training loss as well as the validation loss. This would give you the current training loss without taking the average of all training batches during training and would disable effects of e.g. dropout.<br>
However, you’ll spend time during each training epoch for this calculation, which is usually not needed.</p>
<p>You could run it once in a while to verify that the gap comes from the dropout layers.</p>
Matrix of Dot Products

username: mohammadf
cooked: <p>Given some tensor of input samples of the shape (n_samples, n_features), what’s the best way to calculate a matrix with the shape (n_samples, n_samples) where the element e_{i,j} contains the dot product of the samples with the indices i and j?</p>

username: ptrblck
cooked: <p>I might misunderstand the question, but would</p>
<pre><code class="lang-python">x = torch.randn(5, 10)
torch.matmul(x, x.t())
</code></pre>
<p>work?</p>
Easiest way to get a layer's kernel size, stride, channels, etc...?

username: ProGamerGov
cooked: <p>Like for example from “2d” layers like these:</p>
<pre><code class="lang-auto">Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
</code></pre>
<p>Does PyTorch have any way to extract the information that I desire?</p>

username: ptrblck
cooked: <p>If you just want to have a look, you could just print them:</p>
<pre><code class="lang-auto">conv = nn.Conv2d(
    in_channels=100,
    out_channels=5,
    kernel_size=3,
    stride=1,
    padding=1,
    groups=1)
print(conv)
&gt; Conv2d(100, 5, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
</code></pre>
<p>Otherwise if you need to access them, you can do it directly:</p>
<pre><code class="lang-auto">conv.in_channels
conv.out_channels
conv.kernel_size
conv.stride
conv.padding
conv.groups
</code></pre>
<p>Is this what you are looking for?</p>

username: ProGamerGov
cooked: <blockquote>
<p>Is this what you are looking for?</p>
</blockquote>
<p>Yes! I was trying to print the layers in a way that more closely resembled Lua/Torch7’s printing, like this:</p>
<pre><code class="lang-auto">Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
</code></pre>
<p>To:</p>
<pre><code class="lang-auto">Conv2d(3 -&gt; 64, 3x3, 1,1, 1,1)
</code></pre>
<p>I had a complicated setup with replace and split, before using regex, but things still weren’t working. Hopefully I can make my code a lot more simple and easier to understand now!</p>

username: vosferatu
cooked: <p><a href="https://discuss.pytorch.org/u/ptrblck">ptrblck</a></p>
<p>Is there a way to get the kernel?</p>
<p>For example if I have:</p>
<pre><code class="lang-auto">layer = Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
W = layer.weight
print(W.size())
</code></pre>
<p>The result will be:</p>
<pre><code class="lang-auto">torch.Size([128, 128, 3, 3])
</code></pre>
<p>But I want to save the kernel at the end of training, so only the 3x3 weights to be able to initialize the layer using this kernel in another network with requires_grad=False</p>
<p>How can I do this?</p>

username: ptrblck
cooked: <p>Based on the setup of <code>nn.Conv2d</code> you would use <code>128</code> kernels each with a shape of <code>[in_channels=128, height=3, width=3]</code>. Your layer won’t have a single <code>3x3</code> kernel, so you should instead save the <code>layer.weight</code> tensor (or better save <code>layer.state_dict()</code>).</p>
No module named 'torch.cuda.amp.autocast'

username: Sanjayvarma11
cooked: <p>I am training a model using google colab and i got this error when i am trying to import autocast</p>
<pre><code class="lang-auto">code used to import autocast

import torch.cuda.amp.autocast

error i obtained is
ModuleNotFoundError: No module named 'torch.cuda.amp.autocast'
</code></pre>
<p>Thank you</p>

username: ptrblck
cooked: <p>Answered <a href="https://discuss.pytorch.org/t/basic-autocast-usage/76005/10">here</a>.<br>
Please try to avoid double posts. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>