akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to calculate the multilabel accuracy?

Jake_Eum: <p>Here I try to calculate the multilabel accuracy<br>
is there any part is not correct?</p>
<blockquote>
<p>def validation(test_loader,model):<br>
<span class="hashtag-raw">#print</span>(‘validation’)<br>
tk = tqdm(test_loader, total=len(test_loader),position=0,leave=True)<br>
outputs_ext,targets_ext = <span class="chcklst-box fa fa-square-o fa-fw"></span>,<span class="chcklst-box fa fa-square-o fa-fw"></span><br>
model.eval()<br>
with torch.no_grad():<br>
for i, (images,targets) in enumerate(tk):<br>
device = torch.device(‘cuda’ if torch.cuda.is_available() else ‘cpu’)<br>
images, targets = images.cuda(),targets.cuda()<br>
outputs = model(images)<br>
loss = criterion(outputs,targets)</p>
<pre><code>        #print('outputs-&gt;',outputs[0])
        #print('targets-&gt;',targets[0])
        for out in outputs:
            outputs_ext.extend(out.tolist())
        for tar in targets:
            targets_ext.extend(tar.detach().cpu())
        #oututs_ext = [i for i in np.array(outputs_ext)]
    targets_ext_np = np.array(targets_ext)
    outputs_ext_np = np.array(outputs_ext)
print('outputs_ext--&gt;',outputs_ext_np.round())
print('targets_ext-&gt;',targets_ext_np)

return accuracy_score(outputs_ext_np.round(), targets_ext_np)
</code></pre>
</blockquote>

ptrblck: <p>I don’t know what kind of outputs your model provides, i.e. if they are logits, probabilities, class predictions etc., but <code>outputs_ext_np.round()</code> looks a bit unusual, so I would recommend to double check that it’s indeed working as expected.</p>
Is there a reason to use two optimizer?

Paarulakan: <p>Is there a reason to use two optimizer instead of one for both encoder and decoder in the following tutorial?</p>
<p><a href="http://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html#training-the-model" class="onebox" target="_blank" rel="nofollow noopener">http://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html#training-the-model</a></p>

SimonW: <p>In this case, it is equivalent to use one. But multiple optimizers are useful when you either want different optimization algorithm for different parts or optimize different set of parameters at each time. <img src="//discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ataxias: <p>Separate optimizers will also allow you to have separate schedulers, in case e.g. you are fine-tuning a pretrained model, and you want to slowly ramp-up the learning rate for the low-level (first) part of the model, while the final part will have a higher learning rate from the beginning.</p>
Turning a function into an nn.Module

shapovalov: <p>I see that there are no modules in <code>nn</code> to e.g. add a constant. I understand that it is easy to do in an imperative way, but it feels like there should also be a way to do simple arithmetics within <code>nn.Sequential</code>.<br>
In LuaTorch, there were nn.Add/nn.Mul, etc. Do I miss some obvious way to define a state-less <code>nn.Module</code> in one line? Something like</p>
<pre><code class="lang-auto">nn.Sequential(
    ...,
    nn.ModuleFromFunction(lambda x : x + 2),
)
</code></pre>
<p>to add a constant 2 to the result of the previous operation, where the function passed to <code>nn.ModuleFromFunction</code> implements <code>nn.Module.forward()</code>. Or is it not a common use case?</p>

Naruto-Sasuke: <p><a href="https://pytorch.org/tutorials/beginner/nn_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/nn_tutorial.html</a></p>

shapovalov: <p>Yes, <code>class Lambda</code> from the <code>nn.Sequential</code> example is exactly what I was describing. Is it / should it be included in Pytorch?</p>

ptrblck: <p>That might be a good feature request. Since it’s quite easy to implement, I’m unsure if it’ll land in the core, but I still think it’s worth discussing it.<br>
Would you mind creating this request on <a href="https://github.com/pytorch/pytorch/issues">GitHub</a> and describing your use case etc.?</p>
N dim tensor in pytorch

Arash_Ahmadi: <p>Hi everyone,</p>
<p>I want to make a tensor T of dim n (arbitrary)  which is something like <code>T_{i1,...,in}</code> such that each of i can be 0, 1, or 2 which must be something like <code> T = torch.tensor(3, 3, ..., 3)</code>. Do you have any idea how can I do this?</p>
<p>Many thanks in advance.</p>

ptrblck: <p>I don’t fully understand the use case.<br>
Would you like to set the size as 0, 1, or 2 or the input values?<br>
The former case would be possible via the factory methods, e.g.: <code>torch.randn(3)</code>, while the latter would work via e.g. <code>tensor.fill_(value)</code> or <code>torch.full(...)</code>.</p>

Arash_Ahmadi: <p>Thanks for the help, the use case is a scientific program that I am working on.<br>
BTW, I need to make an n-dimensional tensor so it has n subscript for example<br>
<code>T_ 00...001 </code> is one of them, <code>T_ 0....002 </code>, <code>T_ 0...010 </code> and etc. Naively thinking about this problem is to use n layers of loops, <code>for i in range(3): </code>, but I thought maybe tf or torch can help with this. In the end, I will input some value for each of these T.</p>

ptrblck: <p>Could you post an example, how a potential tensor could look like, i.e. what shape it has, which values it contains, and maybe even the unwanted for loop?</p>

Arash_Ahmadi: <p>Yes of course. let’s take a small n = 2, then I can initialize my tensor with<br>
<code> T = torch.ones(3,3)</code><br>
and I can assign anything I want, as an example:<br>
<code> For i in range(3):</code><br>
<code>For j in range(3):</code><br>
<code>T[i][j] = j*i</code></p>
<p>but if I want to put n = 3;<br>
<code>T = torch.ones(3,3,3)</code><br>
and if I want to do something like what I did for n=2 I need 3 loops.<br>
but what I really want to do is for n= 100 and it is basically impossible this way.</p>
<p>Thanks for spending time on this.</p>

ptrblck: <p>Thanks for the update.<br>
Based on your provided code you could use broadcasting and multiply the input tensor as:</p>
<pre><code class="lang-python">T = torch.ones(3,3)

for i in range(3):
    for j in range(3):
        T[i][j] = j*i

print(T)
&gt; tensor([[0., 0., 0.],
          [0., 1., 2.],
          [0., 2., 4.]])

a = torch.arange(3)
res = a * a.unsqueeze(1)
print(res)
&gt; tensor([[0, 0, 0],
          [0, 1, 2],
          [0, 2, 4]])

a = torch.arange(100)
res = a * a.unsqueeze(1)
print(res)
&gt; tensor([[   0,    0,    0,  ...,    0,    0,    0],
          [   0,    1,    2,  ...,   97,   98,   99],
          [   0,    2,    4,  ...,  194,  196,  198],
          ...,
          [   0,   97,  194,  ..., 9409, 9506, 9603],
          [   0,   98,  196,  ..., 9506, 9604, 9702],
          [   0,   99,  198,  ..., 9603, 9702, 9801]])
</code></pre>
Is a non-automatically-scaling OpenMP thread-pool better than an automatically-scaling one?

imaginary: <p>By default, PyTorch has a fixed-sized OpenMP thread-pool (equal to number of physical cores on a processor). Most PyTorch users who run multiple PyTorch jobs concurrently on CPU probably use <code>taskset</code> and/or <code>numactl</code> to set the particular CPU cores their PyTorch jobs can use, in order to saturate all cores of their system. Doing so also avoids slowdown due to OpenMP synchronization for operations that should not be parallelized.</p>
<p>Let’s consider an hypothetical scenario. Say, a user assigned 8 cores to one of their PyTorch jobs.<br>
If their PyTorch job’s workload also entails operations that are better suited towards being run on fewer cores than they assigned, should they programmatically scale the number of threads before and after such operations with <code>torch.set_num_threads()</code>? As in, would decreasing the number of OpenMP threads before such operations to run such operations on fewer cores and increasing them after those operations be efficient enough to amortize the thread destruction &amp; creation cost? If yes, then a variable-sized thread-pool would automatically handle creating &amp; destroying the number of threads in the OpenMP thread pool.</p>
<p>Talking about <code>parallel_for</code> in PyTorch, due to issue <a href="https://github.com/pytorch/pytorch/issues/24080" rel="noopener nofollow ugc">24080</a>, <a href="https://github.com/pytorch/pytorch/pull/26886" rel="noopener nofollow ugc">PR 26886</a> automatically scaled down the number of threads in the OpenMP thread pool, thus creating a variable-sized thread pool.</p>
<p>Because of issue 32008, the thread-pool was made fixed-sized again with PR 32875. The stated rationale for reverting to the <code>omp parallel if</code> style was to avoid synchronization when <code>num_threads</code> is 1. <strong>However, both <code>omp parallel if</code> &amp; <code>num_threads(num_threads)</code> can be used together, and <code>num_threads(1)</code> doesn’t lead to OpenMP being used anyway. Anyway, was the original rationale (the very first commit of <code>ParallelOpenMP.h</code>) for <code>omp parallel if</code> without using <code>num_threads</code> was to use a fixed-sized thread pool?</strong></p>
<p>Intuitively, it does seem to me that a fixed-sized thread pool would perform better than a variable-sized thread pool if <code>taskset</code> and/or <code>numactl</code> are used to run PyTorch jobs, but the benchmarks on <code>pytorch/benchmarks</code> are not compatible with the latest Python versions. I’m trying to learn about Systems design decisions of various systems &amp; it’d be great to be able to learn why PyTorch initially had a fixed-sized OpenMP thread pool but for a period of time, was okay with a variable-sized thread-pool as well.</p>

imaginary: <p>It turns out that I misunderstood how <code>num_threads</code> works. It is only valid in its scope.</p>
Inverse_transform on user input (Single input)

researchbuddy: <p>I have a dataset with just 4 categorical values<br>
I have used LabelEncoder with OneHotEncoder with one field field43 as y variable or dependent<br>
I get good accuracy of 0.67 with precision also … not bad.<br>
I have train split and test as 0.25</p>
<p>Now when i provide a user input with this small amount of data, I get inverse transform as label not seen. Yes I have given onehotencoder handle unknown - ‘ignore’</p>
<p>Can i get pointers to examples where I can get inverse transform and get prediction? Guidance appreciated. To emphasize … I am looking for example on single user input which gives prediction. Thank you so much. Logan</p>
<p>Just a note: I am using SVM classifier</p>
Templated Cooperative Group Kernel AT Dispatch

5had3z: <p>Has anyone had success with cooperative groups for grid synchronisation in their kernel when implementing new functions?</p>
<p>I’m getting unresolved overload errors when trying to use cudaLaunchCooperativeKernel(), whereas it was previously working fine when launching normally kernel&lt;&lt;&lt;&gt;&gt;&gt;(*args). However, I need grid syncrhonisation during reduction so need to launch a cooperative kernel.</p>
<pre><code class="lang-auto">template &lt;typename scalar_t, std::size_t dims&gt;
using TensorAccR = torch::PackedTensorAccessor32&lt;scalar_t,dims,torch::RestrictPtrTraits&gt;;

template &lt;typename scalar_t&gt;
__global__ void a_kernel(const TensorAccR&lt;scalar_t, 4&gt; in_features, 
TensorAccR&lt;scalar_t, 2&gt; out_features)
{
   cooperative_groups::grid_group g = cooperative_groups::this_grid();
   /*do things*/
   g.sync();
}

void my_launcher(torch::Tensor features_in, torch::Tensor features_out)
{
   AT_DISPATCH_FLOATING_TYPES_AND_HALF(features.scalar_type(), 
   "instance_pool_forward_kernel",
   [&amp;] {
      TensorAccR&lt;scalar_t, 4&gt; features_in_acc = features_in.packed_accessor32&lt;scalar_t,4,torch::RestrictPtrTraits&gt;();
      TensorAccR&lt;scalar_t, 2&gt; features_out_acc = features_out.packed_accessor32&lt;scalar_t,2,torch::RestrictPtrTraits&gt;();

      void *kernelArgs[] = { (void*)&amp;features_in_acc, (void*)&amp;features_out_acc};

      cudaLaunchCooperativeKernel(a_kernel&lt;scalar_t&gt;, gridDims,
                    blockDims, kernelArgs, s_mem, at::cuda::getCurrentCUDAStream());
   });
}</code></pre>

5had3z: <p>Sitting back down after lunch, turns out all was missing was (void*) cast at the function call (shown below), so now it at least compiles fine.</p>
<pre><code class="lang-auto">cudaLaunchCooperativeKernel((void*)a_kernel&lt;scalar_t&gt;, gridDims,
                    blockDims, kernelArgs, s_mem, at::cuda::getCurrentCUDAStream());
</code></pre>
How to do inference with DistributedDataParallel?

asivap: <p>I can successfully train a DDP model for an epoch across several processes. I want to evaluate on a cross validation set after the epoch. If I do this in process 0, it hangs presumably because it’s waiting for synchronization which never comes.</p>
<p>How can I get a normal model from the DDP model? Currently, I’m using a workaround where I save the DDP model state and load it into a temp model. This temp model then evaluates on the validation set.</p>
Issue with torch.mean

chromade: <p>Hello,</p>
<p>I am sure this is a trivial problem <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=9" title=":sweat_smile:" class="emoji" alt=":sweat_smile:">. I am following <a href="https://github.com/jwuphysics/blog/blob/master/_notebooks/2020-08-27-image-attribution-for-galaxies.ipynb" rel="noopener nofollow ugc">a tutorial for grad CAMS</a>  which is a binary classification problem. I have 8 classes so my dimensions are different:</p>
<pre><code class="lang-auto">grad[0]

tensor([0., 1., 0., 0., 0., 0., 0., 0.], device='cuda:0')
</code></pre>
<p>I set my dim to (8,9) which is what I had expected it to be but it didn’t work. If i change the dim to this &lt; or leave it as default from tutorial, I get the following error:</p>
<pre><code class="lang-auto">1

w = grad[0].mean(dim=(1,2), keepdim=True)

2

gradcam_map = (w * act[0]).sum(0).detach().cpu()

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
&lt;ipython-input-79-76f83767f6d3&gt; in &lt;module&gt;
----&gt; 1 w = grad[0].mean(dim=(1,2), keepdim=True)
      2 gradcam_map = (w * act[0]).sum(0).detach().cpu()

IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)
</code></pre>
<p>So what should I set my dim to?</p>
<p>Thanks!</p>

Unity05: <p>Hi,</p>
<p><code>grad[0]</code> only has one dimension. To get the mean over <code>grad[0]</code> use <code>dim=1</code> or <code>dim=0</code> as the error message suggests.</p>
<p>Regards.</p>

chromade: <p>Cheers for the response.<br>
Unfortunately this causes problems with the rest of the tutorial. The author has used dim=(1,2) for what I assume is grad[0] = [0,1].</p>
<p>Any ideas ?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/3/13235c270cad1c6ff59a2aaa432feb568af925f8.png" data-download-href="https://discuss.pytorch.org/uploads/default/13235c270cad1c6ff59a2aaa432feb568af925f8" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/3/13235c270cad1c6ff59a2aaa432feb568af925f8_2_637x500.png" alt="image" data-base62-sha1="2JiPtM61pbc2M8wVriRvrg3QKqA" width="637" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/3/13235c270cad1c6ff59a2aaa432feb568af925f8_2_637x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/3/13235c270cad1c6ff59a2aaa432feb568af925f8_2_955x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/3/13235c270cad1c6ff59a2aaa432feb568af925f8.png 2x" data-dominant-color="E4DDCA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1078×845 148 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Unity05: <p>Hi,</p>
<p>I’ve just checked out the tutorial and I think <code>grad[0]</code> should be<br>
of shape <code>(128, x, x)</code>.</p>
<p>Regards.</p>
Function for getting convolution gradients from layer gradients

rboone: <p>I’m working with a learning rule that doesn’t operate well with autograd.  Given that I have gradients for the output of a convolutional layer, is there a function in pytorch that allows for me to compute the gradients of the weights of a convolutional layer given the inputs and the gradients at the output?</p>
<p>As a simple example, say I have a 4x4 matrix going through a 2x2 convolution with stride 1, padding 0.  The output would be a 3x3 matrix, thus the gradients at the output would be a 3x3 matrix.  Is there a function I can use (avoiding autograd) that can take this 3x3 matrix and the inputs, and give the gradients for the 2x2 weights of the convolution?</p>
CPU vs GPU timing of cuda operations

Germanunkol: <p>Hi everyone,</p>
<p>I’m profiling a large network with <code>torch.autograd.profiler.profile(use_cuda=True)</code>. As far as I can tell, I run everything on the GPU, but still get quite a few operations where the profiler reports much more time spent on the CPU than on the GPU. Some examples are:</p>
<pre><code class="lang-auto">N5torch8autograd13CopyBackwardsE:
     CPU: 87555.573us      GPU: 1532.959us
mse_loss:
     CPU: 1767.147us      GPU: 93.445us
avg_pool3d_backward:
     CPU: 2655.118us      GPU: 183.594us
add:
     CPU: 10984.388us      GPU: 131.836us
CatBackward:
     CPU: 24874.143us      GPU: 13.672us
</code></pre>
<p>I’m expecting that some of this has to do with the lack of synchronization between CPU and GPU (when I call torch.cuda.synchronize(), some values change).<br>
I’ve tried other profilers (nvprof+nvvp) and get similar results.</p>
<p>My question: If this issue is due to the lack of synchronization, how do I get more accurate execution times?<br>
If it is not due to synchronization and the values are correct, are there tricks to reduce the amount of CPU time? I expected most of these functions to be highly parallelized and benefit greatly from GPU kernels…</p>

Germanunkol: <p>It seem the high CPU times were due to the asynchronous execution of the CPU and GPU instructions (see also <a href="https://discuss.pytorch.org/t/profiling-pytorch-pred-cpu-is-reponsible-for-95-of-runtime/9666/3?u=germanunkol">this reply</a>).</p>
<p>To get better values, I ran:</p>
<p><code>CUDA_LAUNCH_BLOCKING=1 python3 profileNetwork.py</code></p>
<p>Now the CPU times for the functions reported above are almost the same as their GPU times.</p>
<p>If I understand correctly, the CUDA_LAUNCH_BLOCKING flag ensures that when a CPU instruction is waiting for a result from the GPU, the waiting time is no longer accumulated into the reported CPU time.</p>

intelaashish: <p>“the CUDA_LAUNCH_BLOCKING flag ensures that when a CPU instruction is waiting for a result from the GPU, the waiting time is no longer accumulated into the reported CPU time.”   ← Where can you find this claim? I do not seem to find any support about this anywhere on forums or discussions.</p>

albanD: <p>You can find info about this in the cuda doc: <a href="https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#concurrent-execution-host-device" class="inline-onebox">Programming Guide :: CUDA Toolkit Documentation</a></p>

intelaashish: <p>I cannot find any such thing on that webpage. Can you please paste a screenshot or copy the exact sentence from that page that claims this → “the CUDA_LAUNCH_BLOCKING flag ensures that when a CPU instruction is waiting for a result from the GPU, the waiting time is no longer accumulated into the reported CPU time.”</p>

albanD: <p>Ho what it says is a bit different:<br>
The cuda call will wait for the execution to finish.<br>
Meaning that no CPU instruction will have to wait for a result anymore.<br>
And so this distortion won’t happen.</p>
How to manipulate individual weights and can't optimize a non-leaf tensor

localh: <p>I have a task that requires me to tweak individual weights but I am struggling to come up with ways to access the weights neatly. The process that I came up with below, for instance, will not work.</p>
<p>Basically, I would like to:</p>
<ol>
<li>Initialize model</li>
<li>Save a copy of all the initial weights.</li>
<li>Run a mini-batch, send loss backward, and run <code>optimizer.step()</code>.</li>
<li>Use step 2 to overwrite randomly selected <code>n</code> backprop updated weights, which requires some sort of indexing.</li>
<li>Perform step 3 and 4 repeatedly until epoch finishes.</li>
</ol>
<p><strong>Edit</strong>: I have updated the workflow a bit, but now I am stuck dealing with the same error.</p>
<pre><code class="lang-auto">import torch
import numpy as np
from torchvision import datasets, transforms
from torch.utils.data import DataLoader
import torch.nn.functional as F

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

# initialize model
class mlp1(torch.nn.Module):
    def __init__(self, num_features, num_hidden, num_classes):
        super(mlp1, self).__init__()
        self.num_classes = num_classes
        self.input_layer = torch.nn.Linear(num_features, num_hidden)
        self.out_layer = torch.nn.Linear(num_hidden, num_classes)

    def forward(self, x):
        x = self.input_layer(x)
        x = torch.sigmoid(x)
        logits = self.out_layer(x)
        probas = torch.softmax(logits, dim=1)
        return logits, probas

# instantiate model
model = mlp1(num_features=28*28, num_hidden=100, num_classes=10).to(device)

# get params
param_optimizer = list(model.named_parameters())
# look for just thse params
params = ['weight', 'bias']
# group them
optimizer_grouped_parameters = [
        {'params': [param for name, param in param_optimizer if any(weight in name for weight in params)] },
        ]

# store indices to freeze
frozen_indices = []
# store frozen weight values
frozen_weights = []
# specify percent to freeze
freeze_percent = 0.7
for i in range(len(optimizer_grouped_parameters[0]['params'])):
    # gen indices
    f_indices = torch.tensor(np.sort(
                                     np.random.choice(np.arange(len(optimizer_grouped_parameters[0]['params'][i].view(-1))),
                                                       size=int(freeze_percent*len(optimizer_grouped_parameters[0]['params'][i].view(-1))),
                                                        replace=False))).long()
    # append indices
    frozen_indices.append(f_indices)

    # retrieve weights to freeze; use a mask for fast access
    f_weights = optimizer_grouped_parameters[0]['params'][i].view(-1)[f_indices]
    # save them to a container
    frozen_weights.append(f_weights)

# replace weights after one mini-batch
for i in range(len(optimizer_grouped_parameters[0]['params'])):
    optimizer_grouped_parameters[0]['params'][i].view(-1)[frozen_indices[i]] = frozen_weights[i]

# optim
optimizer = torch.optim.SGD(optimizer_grouped_parameters, lr=0.1)
# ValueError: can't optimize a non-leaf Tensor


</code></pre>
Conditional cusom autograd question

xvdp: <p>I understand that one can write custom backward() override, I read thru docs and some discussion and I feel a little stupid, the question is not hard, so someone is bound to give me a single line asnwer</p>
<p>I was reading Veit 2016 <a href="https://arxiv.org/pdf/1605.06431.pdf" rel="noopener nofollow ugc">Residual Networks Behave Like Ensembles of Relatively Shallow Networks</a><br>
And I wanted to test the claims (page 6) on other models. But first I need to modify resnet.</p>
<p><em>&gt; To sample a path of length k, we first feed a batch forward through the whole network.  During the backward pass, we randomly sample k residual blocks. For those k blocks, we only propagate through the residual module; for the remaining n−k blocks, we only propagate through the skip connection.</em></p>
<p>So I dont really need to modify any low level computation, just the block, conditionally,<br>
the forward will not change but in some cases the backward should appear to correspond to a shorter forward</p>
<pre><code class="lang-python">    def forward(self, x):
        identity = x
        out = self.conv1(x)
        out = self.bn1(out)
        out = self.relu(out)
        out = self.conv2(out)
        out = self.bn2(out)
        if self.downsample is not None:
            identity = self.downsample(x)
        out += identity
        out = self.relu(out)
        return out

    def run_backwards_on_sometimes_appearing_as_forward(self, x):  
        out = self.relu(x)
        return out
</code></pre>
<p>Seems simple but im blanking out. thanks</p>

xvdp: <p>Maybe as simple as adding a flag to the class, <code>self.mode=0</code> then</p>
<pre><code class="lang-python">   def forward(self, x):
       identity = x
        if self.mode:
            with torch.no_grad():
                out = self.conv1(x)
                out = self.bn1(out)
                out = self.relu(out)
                out = self.conv2(out)
                out = self.bn2(out)
        else:
            out = self.conv1(x)
            out = self.bn1(out)
            out = self.relu(out)
            out = self.conv2(out)
            out = self.bn2(out)
        if self.downsample is not None:
            identity = self.downsample(x)
        out += identity
        out = self.relu1(out)
        return out
</code></pre>
<p>the modified resnet does not fail; i havent checked what gradients are doing yet, but seems should work… if anyone sees any issue flag me. thanks.</p>
Folding conv transpose

zookeeper: <p>Currently only conv+bn folding is happening in graph optimizer. Why not fold convtranspose(deconv) as well as its implementation is essentially convolution, is there some other considerations not to do this? I am suspecting the conv transpose is not implemented the same in different backends…</p>
<p>Thanks!</p>

tom: <p>You chance to be a hero. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
More seriously, the most common thing people want to run on this is image detection using conv, transposed convolution is relatively rare compared to that, so it didn’t get the same attention so far. …until you came along!</p>
<p>Best regards</p>
<p>Thomas</p>

jfix: <p>As you mentioned, it depends entirely on the backend whether it has a fused implementation for the op. So you’d need to add support for it in your backend if you see it has a chance to increase performance. One way to do this is to add your own backend-specific <code>ConvolutionFusedTranspose</code> Node and then look for Convolutions followed by Transposes and replace them in the graph with your own Node, and then map down that Node to your specialized kernel.</p>

zookeeper: <p>Hi jfix, I maybe not clear… I am talking about convtranspose(deconv) + bn. Not exactly conv + transpose…</p>

jfix: <p>Ah – yeah I think someone just needs to implement it <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Floating point exception (core dumped) for the resnet50-v2 model

ravishk1: <p>I tried to compile the model (resnet50-v2-7.onnx) which I downloaded from <a href="https://github.com/onnx/models/tree/master/vision/classification/resnet/model" rel="noopener nofollow ugc">ONNX-ZOO</a>. When I just compiled the model, by command<br>
<em>./model-compiler -backend=CPU -model=resnet50-v2-7.onnx -emit-bundle/ -bundle-api=dynamic</em><br>
It runs perfectly fine, but when I compiled it after creating the profile, by command</p>
<p><em>./image-classifier cat_285.png -image-mode=0to1 -model=resnet50-v2-7.onnx -dump-profile=profile.yml -model-input-name=data</em><br>
<em>./model-compiler -backend=CPU -model=resnet50-v2-7.onnx -emit-bundle=bundle2/ -bundle-api=dynamic -load-profile=profile.yml -relocation-model=pic</em></p>
<p>It produces an error, <strong>Floating point exception (core dumped)</strong></p>
<p>What can be the problem? It’s clear that the profiler is creating the problem in some node, but how this error can be resolved?</p>

jfix: <p>It’s hard to say what the problem could be without more details. Can you compile in debug mode/run in a debugger to get a stack trace?</p>
Error when en2gr is run

Kittu28: <p>I am getting the following error when i run en2gr.<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/9/29bde3608263074dc6470a3db5800978c1f36af8.png" data-download-href="https://discuss.pytorch.org/uploads/default/29bde3608263074dc6470a3db5800978c1f36af8" title="en2gr"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/9/29bde3608263074dc6470a3db5800978c1f36af8_2_597x500.png" alt="en2gr" data-base62-sha1="5XgppvCknmU9RVrJB0S1PdVcWhy" width="597" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/9/29bde3608263074dc6470a3db5800978c1f36af8_2_597x500.png, https://discuss.pytorch.org/uploads/default/original/3X/2/9/29bde3608263074dc6470a3db5800978c1f36af8.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/9/29bde3608263074dc6470a3db5800978c1f36af8.png 2x" data-dominant-color="411F36"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">en2gr</span><span class="informations">731×612 101 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

jfix: <p>It seems like the protobuf isn’t being loaded correctly. Are you sure the model is correctly downloaded and located in the folder <code>glow/utils/en2gr</code> that you’re pointing it to?</p>
Cannot use Dataloader, Type error

Hecate_Astarte: <p>Hello, I am getting this error “TypeError: ‘numpy.float64’ object cannot be interpreted as an integer” when trying to iterate through my DataLoader.</p>
<pre><code class="lang-auto">class MyFramesDataset(Dataset):
    
    def __init__(self, paths, labels, num_frames, transform=None):
        self.video_path = paths
        self.num_of_frames = num_frames
        self.labels = labels
        self.transform = transform

    def __len__(self):
        return(len(self.video_path))
    
    def __getitem__(self, index):
        '''
        Generates one sample of data.
        '''
        label = self.labels[index]
        num_frames = self.num_of_frames[index]  

        X = []
        # Read frames of video specified from video_path 
        for i in range(num_frames):
            image = Image.open(os.path.join(video_path, 'frame{:04d}.jpg'.format(i))).convert('RGB')
            if self.transform:
                image = self.transform(image)
            X.append(image)
            
        X = torch.stack(X, dim=0)
        y = torch.LongTensor(label)    
        return X, y  

transforms = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])

batch_size = 16
train_frame_dataset = MyFramesDataset(train_paths, train_labels, train_num_frames, transforms)
train_dataloader = DataLoader(train_frame_dataset, batch_size, shuffle=False)
</code></pre>
<pre><code class="lang-auto">for batch_idx, (X, y) in enumerate(train_dataloader):
    print(batch_idx)
    print(X,y)
</code></pre>
<p>the error I am getting is: <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/1/6103f6b785041bdafae1f5f9f552c7480958a08f.png" data-download-href="https://discuss.pytorch.org/uploads/default/6103f6b785041bdafae1f5f9f552c7480958a08f" title="help"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/1/6103f6b785041bdafae1f5f9f552c7480958a08f_2_683x500.png" alt="help" data-base62-sha1="dQeO1JtWq0ddeY6pRS8LH52UjWT" width="683" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/1/6103f6b785041bdafae1f5f9f552c7480958a08f_2_683x500.png, https://discuss.pytorch.org/uploads/default/original/3X/6/1/6103f6b785041bdafae1f5f9f552c7480958a08f.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/1/6103f6b785041bdafae1f5f9f552c7480958a08f.png 2x" data-dominant-color="F4F7F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">help</span><span class="informations">790×578 76.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

m-pektas: <p>The reason for this error is the type of “num_frames” variable. Its type is a float but “range” function wants integer. You need to convert the type of “num_frames” from float to int.</p>
Old pytorch 0.4 cuda implementation to pytorch 1.0 questions

Theodorus_Zhou: <p>I encounter a same problem like <a href="https://discuss.pytorch.org/t/converting-old-pre-pytorch1-0-c-cuda-8-extensions-to-pytorch-1-0-c-cuda-9-extension/33862" class="inline-onebox">Converting Old (pre-PyTorch1.0) C++/CUDA 8 extensions to PyTorch 1.0 C++/CUDA 9 extension</a>.</p>
<p>The project I want to convert is here: <a href="https://github.com/speedinghzl/CCNet" rel="nofollow noopener">https://github.com/speedinghzl/CCNet</a></p>
<p>I follow the official guide and write <code>PYBIND11_MODULE</code> in <code>src/lib_cffi.cpp</code> file and a <code>setup.py</code> to compile the whole source files. But it fails and mainly shows the errors below:</p>
<pre><code class="lang-auto">/home/users/USERNAME/anaconda3/lib/python3.6/site-packages/torch/lib/include/pybind11/detail/descr.h:38:42: error: invalid use of incomplete type ‘struct CUstream_st’
         return {{&amp;typeid(Ts)..., nullptr}};
                                          ^

/home/users/USERNAME/anaconda3/lib/python3.6/site-packages/torch/lib/include/pybind11/pybind11.h:168:73: error: ‘static constexpr std::array&lt;const std::type_info*, (sizeof... (Ts) + 1)&gt; pybind11::detail::descr&lt;N, Ts&gt;::types() [with long
 unsigned int N = 67ul; Ts = {CUstream_st}]’ called in a constant expression
         PYBIND11_DESCR_CONSTEXPR auto types = decltype(signature)::types();
</code></pre>
<p>Could any one give me a help? I am very confused about this error and have no idea to solve it.</p>

Theodorus_Zhou: <p>I have solved it successfully. Thanks.</p>

mjoux: <p>This is a bit late, but I’m running into a similar problem. Do you remember how you solved this ? Thanks.<br>
EDIT: mentioning <a class="mention" href="/u/theodorus_zhou">@Theodorus_Zhou</a> explicitly so you might get a notification…</p>

mjoux: <p>Just for reference here: it seems like one cannot expose cuda types directly using pybind. So they have to wrapped by other cpp classes/structs to be able to expose them to python through pybind.</p>
Finetuning self supervised model

nik1: <p>Hello Everyone,</p>
<p>I have a question regarding the use case that I am working on. So, my task is to train AlexNet in self-supervised manner first, by passing the rotated images of the CIFAR10 dataset, and train the model to predict the rotation</p>
<p>After that, I need to extract the features of the first two conv layer of the self-supervised model and use the features to train another model in supervised way on CIFAR10 dataset by just adding a fully connected layer on top of the first two conv layers extracted from self-supervised model. So, currently, I am saving the model parameters and loading the parameters to the Alexnet class. Then, passing the model as a parameter to another class(Alexnet_supervised). I am not sure if this is the correct approach for the same.</p>
<p>Model1 = Alexnet()</p>
<p>checkpoint = torch.load(‘best_model.pt’)</p>
<p>Model1.load_state_dict(checkpoint[‘model_state_dict’])</p>
<p>Model2 = Alexnet_supervised(Model1)</p>
<p>Could anyone help me with this?.</p>
<p>Thanks in advance.</p>

ptrblck: <p>I don’t know, how <code>Alexnet_supervised</code> is implemented, but assuming it’s using the first two layers of <code>Model1</code> and adds a linear layer on top of it, your approach should be correct.</p>

nik1: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a>, Yes I am doing exactly what you said. Using the first two layers of Model1 and adding a linear layer on top of it.<br>
I have another question for the same use case. So, when I train my model for rotation prediction task with Adam optimizer, lr = 0.001, it gives less accuracy around 40%. But, if I change the learning rate to 0.0001, accuracy increases to 84%.<br>
Is this normal for the model to behave like this or there is some bug in my code that I am not able to figure out?<br>
Thanks in advance.</p>

ptrblck: <p>This might be expected, as a high learning rate wouldn’t allow the model to converge and the loss would get stuck at a particular level.<br>
The first figure in e.g. <a href="https://www.jeremyjordan.me/nn-learning-rate/">this post</a> is often used to illustrate this behavior.</p>
Tensor.topk returns a class that doesn't exist

codeknight11: <p>calling tensor.topk should return top_probabilites and top_classes.<br>
This call is returning a class 0 which shouldn’t exist based on category mapping. What am I doing wrong?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/5/05185c24bbc4be6daebfba315f8d6669d2c0c864.png" data-download-href="https://discuss.pytorch.org/uploads/default/05185c24bbc4be6daebfba315f8d6669d2c0c864" title="topk"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/5/05185c24bbc4be6daebfba315f8d6669d2c0c864.png" alt="topk" data-base62-sha1="J4zQSJBXsgcK8LdlS9NAs1B8i0" width="690" height="342" data-dominant-color="F2F0F0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">topk</span><span class="informations">918×456 26.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<pre><code class="lang-auto">def predict(image_path, model):

    tens_im = preproc(image_path)    
    with torch.no_grad():
        tens_im = torch.FloatTensor(tens_im)
        tens_im.unsqueeze_(0)

        probabilities = torch.exp(model.forward(tens_im.to(device)))    
        top_p, top_class = probabilities.topk(5, dim=1)
        top_p = top_p.to("cpu"); top_class = top_class.to("cpu");
        top_p = top_p.numpy()[0]; top_class = top_class.numpy()[0]
        
        return top_p, top_class
</code></pre>

francois-rozet: <p><code>topk</code> returns the k top values and their <strong>indices</strong>. Indices start at 0. So if you have N classes, their respective indices are <code>0, 1, 2, ..., N - 1</code>.</p>
<p>By the way, instead of <code>x.to("cpu")</code> you can use <code>x.cpu()</code>.</p>

codeknight11: <p>So, while mapping the topk class indices with class category mapping, I need to increment the class indices by 1?</p>

francois-rozet: <p>If you wish to have <code>1, 2, ..., N</code> classes, yes! But keep in mind that they won’t correspond with the indices anymore.</p>
at::kByte keeping negative values in float for inputs to model

James_W: <p>I have a cv::Mat which is CV_32S type (also tried CV_8SC3), it stores both positive and negative values.</p>
<p>When convert it to tensor, the values are messed up:</p>
<pre><code class="lang-auto">cout &lt;&lt; in_img &lt;&lt; endl;

auto tensor_image = torch::from_blob(in_img.data, {1, in_img.rows, in_img.cols, 3}, at::kByte);
</code></pre>
<p>The in_img has negative values, while after print out tensor_image, the values were all totally different than in_img. It somehow seems to be scaled to 255 range</p>
<p>the negative values are gone. how do I keep the negative values? When I do this:</p>
<p>auto tensor_image = torch::from_blob(in_img.data, {1, in_img.rows, in_img.cols, 3}, at::kFloat);</p>
<p>and pass this to the forward of the model, I get seg fault <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>
<p>but, when I originally do:</p>
<p>auto tensor_image = torch::from_blob(in_img.data, {1, in_img.rows, in_img.cols, 3}, at::kByte);<br>
tensor_image = tensor_image.toType(at::kFloat);</p>
<p>it works as expected… which is perplexing to me :((</p>

ptrblck: <p><code>kByte</code> is defined as <code>uint8</code>, so the valid range value is <code>[0, 255]</code>, which explains why the values are clipped to this range.<br>
The last approach should also yield values only in this range and should not contain any negative values.</p>
<p>I guess for <code>CV_32S</code> you could use <code>kInt32</code> instead.</p>

John_J_Watson: <p>Thanks a tonne <a class="mention" href="/u/ptrblck">@ptrblck</a> it helped/solved it <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Using CNN in a NER task

Sergio_Zavota: <p>Hi everyone, I am testing some models on a NER task and now I would like to use a CNN.<br>
In particular, I am using a model which takes as input samples with a length of 512 tokens.<br>
These are fed in a BERT model which hidden_size = 768.<br>
The BERT output shape is (batch_size, length, hidden_size) = (16, 512,768).</p>
<p>Now I would like to use a CNN layer which takes in input these values, produce an output passed then in a Linear Layer which in turn outputs values with shape (16, 512, num_tags), where num_tags are the number of classes to predict.</p>
<p>Just recently I approached to CNNs and I am having troubles with the several hyperparametes to choose and the correct dimension to impose.</p>
Different sizes of train-test input - CNN

nikoskot: <p>Hi,<br>
I want to use the “DeepLabV3 ResNet50” model on a medical image segmentation task. I trained the model with images of size 112x112. For testing when I use images of the same size the model works fine but when I resize the images to, let’s say 224x224, the output shows that no part of the input image corresponds to the desired segment. I don’t get any errors about the input size though. I have read that FCNNs are capable of accepting inputs of different sizes due to the lack of fully connected layers and i have checked that my model has no fully connected layers.</p>
<p>Do CNNs in general “learn” to identify the target objects only on the size they have been trained on?<br>
Does it have to do with the size of the image, the scale of the target object inside the image or both?</p>
<p>Thank you.</p>

Dwight_Foster: <p>Yes if you train a CNN on a certain size image the weights of the model will correspond only to that size image. So if you change the shape in testing the model will not be able to predict it correctly because the weights will not match up with the size of the image but you won’t get an error.</p>
How to create custom dataset with following function?

vishalthengane: <p>I want to create a custom dataset where I can read a CSV file and do the following lambda function. I have created a part of the class but I am getting an error for the <code>map</code> and <code>concatenate</code> function.<br>
Errors:</p>
<pre><code class="lang-auto">AttributeError: 'ConcatDataset' object has no attribute 'map'
AttributeError: 'map' object has no attribute 'map'
</code></pre>
<pre><code class="lang-auto">This is my dataset class.
class MyDataset(Dataset):
    def __init__(self, csv_path):
        self.df = pd.read_csv(csv_path, index_col=False)

    def __getitem__(self, index):
        return self.df.iloc[index, :].to_dict()

    def __len__(self):
        return len(self.df)

    def map(self, func):
        return map(func, self)

    def concatenate(self, dataset):
        # cocatenate dataset
</code></pre>
<p>I want to do the following;</p>
<pre><code class="lang-auto">mydataset = MyDataset("train_data.csv")
datasets = []
for k in range(5):
    datasets.append(mydataset.map(lambda x: some_function)

dataset = datasets[-1]
for d in datasets[:-1]:
    dataset = dataset.concatenate(d)
</code></pre>
<p>Also,</p>
<pre><code class="lang-auto"># again do mapping 
dataset = dataset.map(lambda x: x ** 2)
# and also some other functions like
dataset.repeat()
dataset.filter()

</code></pre>
<p>From what I understood the error is coming because the function is not returning the object of the same class.</p>
<p>Thanks in advance!</p>
Add layer to Faster RCNN

Stylianos_Charalambo: <p>Hello, I am trying to add an extra layer to the backbone of the Faster RCNN model (ResNet 50) but I am having trouble with the tensor shapes if anyone can help me. The code is as below, and I am trying to determine the meta_array_shape and backbone_output_shape variables. The first one is regarding the 2nd input which is a tensor of 3 columns, while the second is trying to match the output of the backbone so that they can be concatenated together.</p>
<pre><code class="lang-auto">        meta_array_shape = 3 
        backbone_output_shape = (512,3)
        
        self.meta_model = nn.Sequential(
            nn.Linear(meta_array_shape, backbone_output_shape),
            nn.BatchNorm1d(backbone_output_shape),
            nn.ReLU(),
            nn.Dropout(p=0.5)
        )
</code></pre>
<pre><code class="lang-auto">        X = self.faster_rcnn.backbone(X)

        # Metadata Net
        Y = self.meta_model(meta_in)
        
        # Concat Feature Maps
        X = torch.cat(X, Y)
</code></pre>

ksz: <p>What is <strong>meta_in</strong>? What shape does it have?</p>
<p>The <strong>backbone_output_shape = (512,3)</strong> determines 512 as batch_size and 3 as output.<br>
The meta_in should be <strong>(512, ?)</strong>, so batch_size should be the same and then:</p>
<pre><code class="lang-auto">nn.Linear(?, 3),
nn.BatchNorm1d(3),
</code></pre>
<p>Where <strong>?</strong> is any number.</p>

Stylianos_Charalambo: <p>Sorry, meta_in is the 2nd input tensor I mentioned above which has 3 columns. The backbone_output_shape is the original output of the faster rcnn model backbone which I am not sure how to obtain</p>

ksz: <pre><code class="lang-auto">import torch.nn

meta_array_shape = 3 
backbone_output_shape = (512,3)

meta_model = nn.Sequential(
    nn.Linear(meta_array_shape, backbone_output_shape[1]),
    nn.BatchNorm1d(backbone_output_shape[1]),
    nn.ReLU(),
    nn.Dropout(p=0.5)
)

X = torch.rand(backbone_output_shape)
Y = meta_model(torch.rand(backbone_output_shape[0], meta_array_shape))
print(torch.cat((X, Y), dim=1).shape)
</code></pre>
<p>or</p>
<pre><code class="lang-auto">import torch.nn

meta_array_shape = 3 
backbone_output_shape = (512,3)

meta_model = nn.Sequential(
    nn.Linear(meta_array_shape, backbone_output_shape[1]),
    nn.BatchNorm1d(backbone_output_shape[1]),
    nn.ReLU(),
    nn.Dropout(p=0.5)
)

X = torch.rand(backbone_output_shape)

Y = torch.rand(meta_array_shape)
Y = Y.repeat((backbone_output_shape[0], 1))
Y = meta_model(Y)

print(torch.cat((X, Y), dim=1).shape)
</code></pre>
<p>If these solutions are not good for you, then please describe the problem in more detail.</p>

Stylianos_Charalambo: <p>The second part is the forward function</p>
<pre><code class="lang-auto">    def forward(self, images_in, meta_in):
        X = self.faster_rcnn.transform(images_in)
        X = self.faster_rcnn.backbone(X)
        Y = self.meta_model(meta_in)
        
        X = torch.cat(X, Y)
        
        ....
                
        return X
</code></pre>