akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
CrossEntropyLoss on subset of output

username: Avi_Chapman
cooked: <p>Hi all,</p>
<p>I’m working on what is effectively an image segmentation problem. However, the majority of the pixels have a category that of ‘I don’t care’. By this I mean that I don’t want the weights affected one way or the other no matter what category those pixels are sorted into.</p>
<p>The pixels that I actually care about are disjoint and represent only a small percentage of the total pixels in the output.</p>
<p>Is there a way to perform CrossEntropyLoss on those scattered disjoint pixels only, while ignoring all of the other pixels? So far, the only sure-fire method I’ve thought of is to visit each pixel of interest and calculate the loss and then add all of the losses together.</p>
<p>Is there a better way?</p>
<p>Thanks in advance!</p>

username: ptrblck
cooked: <p>If you know the position of the irrelevant pixels,  you could create a mask with zeros for them (and ones for the valid locations), and multiply your unreduced loss with it (<code>reduction='none'</code> for the loss function initialization). Afterwards you could calculate the mean (or normalize the loss somehow) and call <code>.backward()</code> as usual.</p>

username: Avi_Chapman
cooked: <p>I tried something similar already, except I applied the mask to the candidate output before putting it through the loss calculation. (The target already had zero at those locations.)</p>
<p>It failed to improve with training, perhaps due to the non-differentiable nature of the operation. I’ll try doing it to the loss result like you are suggesting and see if that works better.</p>
<p>Thanks,</p>

username: KFrank
cooked: <p>Hi Avi!</p>
<p>In addition to <a class="mention" href="/u/ptrblck">@ptrblck</a>’s suggestion, you could try using the<br>
class-weights feature of <code>CrossEntropyLoss</code>.  This allows to<br>
to give each class its own weight in the overall loss.  For your<br>
use case I would imagine setting the weight of your “I don’t care”<br>
class to zero and the weights of your “real” classes to one.</p>
<p>Note, both this and <a class="mention" href="/u/ptrblck">@ptrblck</a>’s approach entail <em>calculating</em> the<br>
loss for all of the pixels, and then discarding most of the calculated<br>
losses (by multiplying them by zero).  This may seem inefficient<br>
if most of your pixels are “I don’t care,” but, presumably, when an<br>
image is pumped through your network, (almost) all of the input<br>
pixels affect the per-pixel predictions for the “real” pixels, so the<br>
overall network processing dominates the cost of calculating the<br>
loss.</p>
<p>Good luck.</p>
<p>K. Frank</p>

username: Avi_Chapman
cooked: <p>Thanks. Another great idea.</p>
<p>Is that similar to using the ignore_index option (which I just discovered)?</p>
How to get cuda cudnn nccl build version?

username: hasakii
cooked: <p>When build from source or install from anaconda channel, we would like to know the exact version of CUDA, CUDNN and NCCL. How could we do that?</p>

username: hasakii
cooked: <p>CUDNN version: <a href="https://github.com/pytorch/pytorch/issues/698#issuecomment-326806780" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/698#issuecomment-326806780</a><br>
<code>torch.backends.cudnn.version()</code></p>
<p>CUDA version: <a href="https://discuss.pytorch.org/t/pytorch-0-4-get-cuda-version/19058/2" class="inline-onebox">Pytorch 0.4 get cuda version</a><br>
<code>torch.version.cuda</code></p>
<p>NCCL version: <a href="https://discuss.pytorch.org/t/how-to-check-if-nccl-is-installed-correctly-and-can-be-used-by-pytorch/26646/2" class="inline-onebox">How to check if NCCL is installed correctly and can be used by PyTorch?</a><br>
<code>torch.cuda.nccl.version()</code></p>
Visualization tool for report

username: oboul
cooked: <p>what would be the best accuracy and loss  visualization tool for writing reports instead of tensorboardX ?</p>

username: albanD
cooked: <p>Hi,</p>
<p>If you want to stay with tensorboard, there now is a builtin Tensorboard support in pytorch, see <a href="https://pytorch.org/docs/stable/tensorboard.html?highlight=tensorboard" rel="nofollow noopener">here</a>.<br>
Otherwise, you can look into visdom which is another visualization project similar to Tensorboard.</p>

username: oboul
cooked: <p>Hi, I mean some thing that looks better then tensorboard<br>
some thing to export as jpg and put into report or poster, or can you download tensorboardX scalars into images ?</p>

username: Nikronic
cooked: <p>you can extract media from tensorboard reports. By the way, Facebook’s Captum is also great tool for better understand. Give it a try.</p>
How to organize image sequences for 3D CNN

username: ajhynes7
cooked: <p>I am working on the segmentation of a video of a person walking. My goal is to segment the body parts of the person.</p>
<p>I have used a U-Net model to perform segmentation on each individual frame. It works well for segmenting the person from the background, but not for segmenting their individual parts. I want to see if I can improve results by leveraging the temporal order of the images.</p>
<p>I’m wondering how to organize the images into sequences for a 3D CNN. Should I use the same number of images in each sequence, say five? Then I could have a batch of 50 sequences each containing five consecutive images. Or would it be better to randomly select the number of images for each sequence? Also, should I shuffle the sequences in the batch?</p>

username: donCalculator
cooked: <p>I don’t think you should shuffle the images if you want to leverage their temporal order. By shuffle I mean in the 3D axis, so I would load images like this batch 0, [Id, Id+1 …], 3, 256, 1024. But if train with more than one batch size than I would shuffle on the 0 axis while training.</p>
<p>Changing the 2D layers to 3D you will increase the number of parameters by about a factor of 3. Make sure you have enough training data before you try.</p>
<p>Also, look at keypoint rcnn first, it might be enough for your application.</p>
Using PyTorch to Build Relative Histograms of Text Data

username: RecursiveReg
cooked: <p>Hello all,</p>
<p>I am entirely new to PyTorch but I am familiar with machine learning from a conceptual perspective. I would like to use PyTorch for a text classification program. Basically, I have lists of words from several documents. I would like to make these lists in to relative histograms for each document where each histogram entry is the number of times each word appears in the document. Since I would like to use each histogram entry as a dimension, I would like entries of 0 occurrences to be made when a word that appears in another document/documents does not appear in the document for which the histogram is associated with.</p>
<p>From there, I would like to use Manhattan distance to compare test documents. How would I go about getting started with this? I don’t expect this to be spoonfed to me, but I have found the documentation to be confusing enough that I am not really sure where to begin with this. Can I even build the histograms using PyTorch or do I have to build them on my own and then pass them off to PyTorch?</p>
<p>Thank you in advance.</p>
<p><strong>Update:</strong> Based on <a href="https://discuss.pytorch.org/t/histogram-function-in-pytorch/5350">this post</a>, it sounds like torch.histc can be used  to do what I want. Is that correct?</p>
Cuboid and obersvation Angle

username: princefr
cooked: <p>Hi.<br>
I’m writing a class Object to help me work with (3D SIDOD Dataset)[<a href="https://research.nvidia.com/publication/2019-06_SIDOD%3A-A-Synthetic" rel="nofollow noopener">https://research.nvidia.com/publication/2019-06_SIDOD%3A-A-Synthetic</a>].</p>
<p>Each object of the dataset has:</p>
<ul>
<li>class</li>
<li>visibility</li>
<li>location (3* 1) representing the translation</li>
<li>quaternion (4* 1)</li>
<li>pose transform (3<em>3 rotation matrix + 3</em>1 translation)</li>
<li>cuboid_centroid (3*1)</li>
<li>projected_cuboid_centroid (2* 1)</li>
<li>bouding box (4*1)</li>
<li>cuboid (8 * 3)</li>
<li>projected cuboid (8 * 2)</li>
</ul>
<p>Has it would result in a long training session (3 or 4 days training session with my GPU till convergence) to train a model to recognize the 9 keypoints of objects (8 corners projected points + centroid points)  i would like to use an another technique to do the 3D task, especially the method that train a model to predict the orientation and the dimension of the object.</p>
<p>i can easily recover the dimension of the object with the cuboid (8 * 3).</p>
<pre><code class="lang-auto">    def getDimensionFrom3D(self):

        ''' corners: (8,3) no assumption on axis direction
                1 -------- 0
               /|         /|
              2 -------- 3 .
              | |        | |
              . 5 -------- 4
              |/         |/
              6 -------- 7
         '''

        l = np.sqrt(np.sum((self.cuboid[0, :] - self.cuboid[1, :]) ** 2))
        w = np.sqrt(np.sum((self.cuboid[1, :] - self.cuboid[2, :]) ** 2))
        h = np.sqrt(np.sum((self.cuboid[0, :] - self.cuboid[4, :]) ** 2))

        return [l, w, h]
</code></pre>
<p>but i’m struggling to understand how can i recover the observation angle using informations i have. (the same that kitti dataset provide).<br>
Anyone can help me ?</p>

username: DerekGloudemans
cooked: <p>Do you have information on the camera calibration intrinsics? If so, you can project the points in the image space into 3D space relative to the camera and calculate the viewing angle there using trigonometry. The OpenCV library should have most of what you need to do this.</p>
Jigsaw problem IndexError: invalid index of a 0-dim tensor. Use tensor.item() to convert a 0-dim tensor to a Python number

username: Niraj_l
cooked: <p>Hi<br>
I am solving Jigsaw Problem, for that I am using this <a href="https://research.wmz.ninja/attachments/articles/2018/03/jigsaw_cifar100.html" rel="nofollow noopener">code</a></p>
<p>but the issue I am facing while using this code is that I am getting error</p>
<p>IndexError: invalid index of a 0-dim tensor. Use tensor.item() to convert a 0-dim tensor to a Python number</p>
<p>Also sharing colab links for quick reference:<br>
<a href="https://colab.research.google.com/drive/1QfQvqZGeKnQpvN8vtj0Vpe7OKt3blvgI" class="onebox" target="_blank" rel="nofollow noopener">https://colab.research.google.com/drive/1QfQvqZGeKnQpvN8vtj0Vpe7OKt3blvgI</a></p>
<p>Many Thanks in advance!</p>

username: albanD
cooked: <p>Hi,</p>
<p>From the error in the colab, you use <code>.data[0]</code>.<br>
<code>.data</code> should not be used anymore.<br>
If the goal here is to get a python number. you should replace <code>.data[0]</code> by just <code>.item()</code> (as suggested in the error message).</p>

username: Niraj_l
cooked: <p>Many thanks <a class="mention" href="/u/alband">@albanD</a> but after doing this I am facing this error,</p>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [32, 16]], which is output 0 of SigmoidBackward, is at version 10; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).</p>
<p>You can check <a href="https://colab.research.google.com/drive/1QfQvqZGeKnQpvN8vtj0Vpe7OKt3blvgI" rel="nofollow noopener">notebook</a></p>

username: albanD
cooked: <p>This is what this error message means:</p>
<blockquote>
<p>one of the variables needed for gradient computation has been modified by an inplace operation</p>
</blockquote>
<p>This means that you did an inplace operation on a Tensor whose original value was required to be able to compute gradients.<br>
You need to remove this inplace operation to be able to compute gradients.</p>
<blockquote>
<p>which is output 0 of SigmoidBackward,</p>
</blockquote>
<p>The problematic Tensor is the output of Sigmoid.</p>
<p>So you need to find where you do an inplace operation on the output of the sigmoid and replace that with an out of place op. Or <code>.clone()</code> the output of the sigmoid.</p>
Parallel For Loop for parallelized sub computation in a gradient step

username: MushroomHunting
cooked: <p>Hi,</p>
<p>I have a simple problem (hopefully!) regarding parallelizing a part of my model. I have looked around but cannot seem to find a definitive answer on how to approach this although similar questions seem to have been asked.</p>
<p>TLDR: How do you do a <em>parallel for loop</em> across multiple CPUs or GPUs in the same computer in the middle of a gradient step?</p>
<p>What I have is multiple additional computations which I know are embarassingly parallelizable, but compute bound. Currently, I am calculating them sequentially in a for loop within a .fit() function. These results are accumulated and then combined to produce the final loss.</p>
<p>A code sketch is as follows:</p>
<pre><code class="lang-auto">for i in range(epochs):  
     self.optimizer.zero_grad()
     loss = self.fit_get_loss()
     loss.backward(retain_graph=False)

def fit_get_loss():
     # This is the for loop to parallelize
     total_loss = 0
     for j in range(self.N_extra_models):
         m1 = self.extra_models_1[j]
         m2 = self.extra_models_2[j]
         loss1 = m1.fit_get_loss(with_grad=True)
         loss2 = m2.fit_get_loss(with_grad=True)
         total_loss = total_loss + loss1 + loss2
     return total_loss

</code></pre>
<p>I would like to distribute the computation across many CPUs (e.g. a workstation with 20 cores) such that , for example, i have 20 of those j iterations occuring in parallel and I just accumulate the loss value. This is preferrable to do on CPU given the hardware I currently have available. However I am also keen to know how to apply the sample problem to multiple GPUs.</p>
<p>Actually, the  real problem I have is more complicated than the above, and I need to re-use the extra_models but I think the above is the simplest form of what i’m trying to achieve. The extension of the problem is that I would like to access the updated fitted values for each of those extra_models within the main optimization loop call.</p>
<p>thanks again</p>

username: albanD
cooked: <p>Hi,</p>
<p>You can use python’s native threading to achieve this. I am not very familiar with it <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>
<p>Keep in mind though that if you perform big enough operations in pytorch’s operations, it will use multiple threads automatically.<br>
If you just have many many python code to run. multiple threads won’t help you, because you can only run one thread at a time in python (you can google for the GIL).</p>
Backward is not being called in custom torch.autograd.Function

username: jpainam
cooked: <p>Hi, I have a custom <code>torch.autograd.Function</code> defined like this.</p>
<pre><code class="lang-python">class SourceMemory(Function):
    def __init__(self, M, alpha=0.01):
        super(SourceMemory, self).__init__()
        self.M = M
        self.alpha = alpha

    def forward(self, inputs, targets):
        self.save_for_backward(inputs, targets)
        outputs = inputs.mm(self.M.t())
        return outputs

    def backward(self, grad_outputs):
        inputs, targets = self.saved_tensors
        grad_inputs = None
        if self.needs_input_grad[0]:
            grad_inputs = grad_outputs.mm(self.M)
        for x, y in zip(inputs, targets):
            self.M[y] = self.alpha * self.M[y] + (1. - self.alpha) * x
            self.M[y] = F.normalize(self.M[y], p=2, dim=0)
        return grad_inputs, None


class IntraNet(nn.Module):
    def __init__(self, beta=0.05, alpha=0.01, num_classes=0, num_features=0, weight=None):
        super(IntraNet, self).__init__()
        self.beta = beta
        self.alpha = alpha
        self.weight = weight
        self.register_buffer('M', torch.zeros(num_classes, num_features))

    def forward(self, inputs, targets, features=None, epoch=None):
        self.alpha = self.alpha * epoch
        outputs = SourceMemory(self.M, alpha=self.alpha)(features, targets)
        outputs /= self.beta
        loss = F.cross_entropy(inputs, targets, weight=self.weight)
        return loss, outputs
</code></pre>
<p><code>M</code> is a <code>memory</code> containing the feature. The IntraNet Module is used as a criterion in</p>
<pre><code class="lang-python">  for i, inputs in enumerate(data_loader):
            inputs, pids = self._parse_data(inputs)
            outputs, features = self.model(inputs)
            # this will update the buffer or the memory M
            source_pid_loss, _ = self.inter_criterion(outputs, pids, features=features, epoch=epoch)
</code></pre>
<p>Where <code>self.inter_criterion</code> is the <code>IntraNet</code> module. and the model returns a <code>output = [batch_size, num_class]</code> and <code>features = [batch_size, 4096]</code>.<br>
The problem is that, <code>backward</code> of <code>SourceMemory</code> is never called. Am I doing something wrong here? I want the buffer <code>M</code> to be filled during the backward pass.<br>
Thank you</p>

username: albanD
cooked: <p>Hi,</p>
<p>You are using “old style” autograd.Function and this can happen.<br>
Can you change your code to use the new style as described in <a href="https://pytorch.org/docs/stable/notes/extending.html#extending-torch-autograd" rel="nofollow noopener">this doc</a> (basically remove the init and pass arguments directly to the forward, make forward and backward static method, save stuff in <code>ctx.</code> during the forward if you need them in the backward) and see if it solves the issue.</p>
Beginner : input and target data in pytorch?

username: astri
cooked: <p>hello all,<br>
i am beginner in deep learning and try learning pytorch. I have two files csv<br>
x : input file : list with (1934,1024) shape<br>
y : target file : list with (1934,8) shape</p>
<p>how i prepare this dataset for feed in my model in pytorch?</p>

username: albanD
cooked: <p>Hi,</p>
<p>You need to load the csv into a Tensor. You can look online how people usually do that.<br>
You can also check the dataloading tutorial for pytorch <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" rel="nofollow noopener">here</a></p>

username: astri
cooked: <p>thanks for your reply , i `ll try to understand from the tutorial.I’ve managed to input and change it to tensor. THe next step is DATA loader is that right? One more thing<br>
I want to ask even i dont have validation data i have to create it with torch.utils.data right?</p>

username: albanD
cooked: <p>You don’t have to. But it will make your code cleaner and easier to extend <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Visualization of the weights of a linear layer

username: cherepanovic
cooked: <p>How to visualize the <strong>weights</strong> and also the <strong>gradients</strong> of a linear layer for example nn.Linear(50, 10) in a proper way for an analysis?</p>
<p>Thanks a lot!</p>

username: Mendel123
cooked: <p>Maybe you can use <code>histgram</code> function in <code>tensorboard</code> to visualize it.</p>

username: cherepanovic
cooked: <p>I do it this way also.</p>
<p>I wonder whether there are also other variations which could provide a better overview…</p>
Lighter conv backbone for faster-rcnn

username: vladsb94
cooked: <p>Hi,</p>
<p>I am am attempting to train a faster-rcnn model using the method presented here: <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html</a>. I would like to use transfer learning considering my data set size is very small: ~250. I am wondering, is there a lighter architecture similar to fasterrcnn_resnet50_fpn with pre-trained weights that I could use for my case? Thanks in advance.</p>
TypeError: img should be PIL Image. Got <class 'numpy.ndarray'>

username: Preet_Khaturia
cooked: <p>Train_loc = ‘./Mehsana/Train_data/’<br>
Val_loc   = ‘./Mehsana/Val_data/’</p>
<p>X = glob.glob(Train_loc+‘Images/’+‘<em>img’)<br>
Y = glob.glob(Train_loc +‘labels/’+’</em>.img’)</p>
<p>X_train, Y_train, X_test, Y_test = X[:2000], Y[:2000], X[2000:], Y[2000:]</p>
<p>del X,Y</p>
<p>class CustomDataset(Dataset):<br>
def <strong>init</strong>(self, image_paths, target_paths, train=True):   # initial logic happens like transform</p>
<pre><code>    self.image_paths = image_paths
    self.target_paths = target_paths
   
    self.transforms  = transforms.Compose([
#transforms.RandomHorizontalFlip(p=0.5),
#transforms.RandomVerticalFlip(p=0.5), 
transforms.Resize((224,224)), transforms.ToTensor(),
transforms.Normalize(
</code></pre>
<p>mean= [688.8383261571815, 890.7605253921796, 1012.9213980828494, 3228.524152857917],<br>
std=[238.88659552814724, 267.57694692565684, 347.344288531034, 689.5044531578629])])</p>
<pre><code>def __getitem__(self, index):

    img =gdal.Open(self.image_paths[index],gdal.GA_ReadOnly)
    img= img.ReadAsArray()
    print(img.shape)
    img = np.moveaxis(img,0,-1)
 
    mask = gdal.Open(self.target_paths[index],gdal.GA_ReadOnly)
    print(np.unique(mask))
    t_image = self.transforms(img)
    return t_image, mask

def __len__(self):  # return count of sample we have

    return len(self.image_paths)
</code></pre>
<p>train_dataset = CustomDataset( X_train, Y_train, train=True)</p>
<p>train_dataset = CustomDataset(train_image_paths, train_mask_paths, train=True)</p>
<p>I get this error when I am trying to generate a custom dataset in pytorch for Unet model. My images are of size (250,250,4).</p>

username: roaffix
cooked: <p>You need to add <code>transfroms.ToTensor()</code> at the end of transformations block and custom DataLoaders require operations with PIL library. So, you can do transformations with numpy and then convert the image to PIL format. In your case, you need to convert the result of <code>img = np.moveaxis(img,0,-1)</code> to PIL and then pass it to transform function.</p>
I cannot find FilterDescriptor::set() in libtorch.so

username: da_yan
cooked: <p>I cannot find at::native::FilterDescriptor::set(at::Tensor const&amp;, long) in the libtorch.so file.</p>
<p>The linker gives error message:</p>
<pre><code class="lang-auto">undefined reference to `at::native::FilterDescriptor::set(at::Tensor const&amp;, long)'
</code></pre>
<p>But I do find this function in pytorch/aten/src/ATen/cudnn/Descriptors.cpp</p>
<p>What’s the problem?</p>
Using main RAM instead of VRAM

username: FR1
cooked: <p>Hi!</p>
<p>How can one use their main desktop’s RAM instead of video board based (GPU) RAM?</p>
<p>Any decreases in speed is not a problem.</p>
<p>Is it a simple PyTorch (or other similar framework) code change, or is it simply not compatible?</p>
<p>Thank you</p>

username: ptrblck
cooked: <p>If you don’t push your model parameters and data to the GPU, all operations will be executed on the CPU by default and will use the RAM.</p>

username: FR1
cooked: <p>Thank you. Is there any way to run the model on the GPU, but store the data in main memory instead of VRAM?</p>

username: ptrblck
cooked: <p>Since the calculations should be performed on the GPU, you would have to push the parameters and data to the device at some point.<br>
Are you running out of memory on the GPU?<br>
If so, you could try to lower your batch size or use <code>torch.utils.checkpoint</code> to trade compute for memory.</p>
Why Does Dataset class Inheritance not require the Super Constructor

username: arif_saeed
cooked: <p>In the pytorch tutorial for the dataloader:<br>
<a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/data_loading_tutorial.html</a></p>
<p>When the Dataset class is inherited by the FaceLandmarksDataset class below.<br>
Super is not used to initialize the Dataset class.<br>
Why is this?<br>
I dont understand how inheritance of the Dataset class works if the Dataset class is not initialized with the Super constructor.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/7/e7aa590a17c744fa215623e79f3f9f39c5b29d94.png" data-download-href="https://discuss.pytorch.org/uploads/default/e7aa590a17c744fa215623e79f3f9f39c5b29d94" title="dataloader"><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/7/e7aa590a17c744fa215623e79f3f9f39c5b29d94.png" alt="dataloader" data-base62-sha1="x3pgnu3AdcpN6Vsj6ukkjgG7r00" width="529" height="500" data-dominant-color="EFEEF1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">dataloader</span><span class="informations">744×702 26.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: Nathan_Drake
cooked: <pre><code class="lang-auto">class parent:
    def plus_one(self, x):
        return x+1

class child(parent):
    pass

instance = child()
instance.plus_one(1)
</code></pre>
<blockquote>
<p>2</p>
</blockquote>
<p>As you can see, the child class dose not have any method. It just inherits its parent class.</p>
Can't gdb in fused_kernel.cpp

username: alanzhai219
cooked: <p>I am looking into PyTorch-jit-fusion now. I try to set a breakpoint in fuser_kernel.cpp at FusedKernelCPU::FusedKernelCPU(…) when I run a jit-trace stript. Anyone can tell me when and how such function will be invoked? Any reply will be appreicated. thanks</p>

username: tom
cooked: <p>Last I looked, CPU fusion was disabled by default, so you need to enable it with<br>
<code>torch._C._jit_override_can_fuse_on_cpu(True)</code>.</p>
<p>Printing <code>myfn.graph_for(inp)</code> will show if you have fusion for your inputs.<br>
Note that as optimized graphs are cached, you need to re-define the traced/scripted function in order to clear the cache between after enabling CPU fusion if you previously ran it with CPU fusion disabled.</p>
<p>Note that CPU fusion is disabled by default due to performance &amp; flakiness issues. In turn AVX &amp; co are disabled in the CPU fuser because they sometimes cause problems (see the commentary in <code>fused_kernel.cpp</code>).</p>
<p>Best regards</p>
<p>Thomas</p>
How to perform cross-correlation

username: sharjeel
cooked: <p>I really need help with cross-correlation. From pytorch docs i saw that conv2d layer can be used for cross-correlation, but when i tried to do it i keep on getting errors and cant figure out how to use conv2d layers for cross-correlation to find a template object in a search region. Could someone help me with the code.</p>

username: ptrblck
cooked: <p>Could you post the code snippet you’ve already written so that we can assist in debugging?<br>
<code>F.conv2d</code> should work with a custom <code>weight</code> parameter as your kernel.</p>

username: sharjeel
cooked: <p>here is the code:<br>
import torch<br>
import torch.nn as nn<br>
import torch.nn.functional as F<br>
import cv2<br>
import numpy as np<br>
import matplotlib.pyplot as plt</p>
<p>IMG_SIZE = 100</p>
<p>image = cv2.imread(“C:\Users\Dutchfoundation\Desktop\AI\Torch\image_01.jpg”,cv2.IMREAD_GRAYSCALE)<br>
template = cv2.imread(“C:\Users\Dutchfoundation\Desktop\AI\Torch\template_01.jpg”,cv2.IMREAD_GRAYSCALE)</p>
<p>image = cv2.resize(image, (IMG_SIZE, IMG_SIZE))<br>
template = cv2.resize(template, (IMG_SIZE, IMG_SIZE))</p>
<p>print(“Image:”,image.shape)<br>
print(“Template:”,template.shape)</p>
<p>image_tensor = torch.Tensor(image).view(-1,100,100)<br>
template_tensor = torch.Tensor(template).view(-1,100,100)<br>
print(“Tensor Image:”,image_tensor.shape)<br>
print(“Tensor Template:”,template_tensor.shape)<br>
image_final = image_tensor.view(-1,1,100,100)<br>
template_final = template_tensor.view(-1,1,100,100)<br>
print(“Final Image:”,image_final.shape)<br>
print(“Final Template:”,template_final.shape)</p>
<p>corr_map = F.conv2d(image_final, template_final)<br>
print(corr_map.shape)</p>
<p><span class="hashtag">#plt</span>.imshow(corr_map)</p>
<p>The output shape of the corr_map is [1,1,1,1] which does’nt make sense</p>

username: ptrblck
cooked: <p>You are using an image and filter the same shape (<code>100x100</code>), which will create a single pixel output.<br>
This is expected in a cross-correlation as well as convolution.<br>
If you want a bigger output shape, use a smaller kernel or a larger image input.</p>

username: sharjeel
cooked: <p>Thanks a lot for the help</p>
Regarding (3 * embed_dim) in nn.MultiheadAttention

username: vainaijr
cooked: <p>why is this 3 * embed_dim in implementation of nn.MultiheadAttention?<br>
what does it mean?</p>
<pre><code class="lang-auto">self.in_proj_weight = Parameter(torch.empty(3 * embed_dim, embed_dim))
</code></pre>
<p>from what I understand, when I do</p>
<pre><code class="lang-auto">x = nn.MultiheadAttention(embed_dim, number_of_heads)
</code></pre>
<p>then, we have Q, K, V which are representations of words in embedding form.<br>
in case of MultiheadAttention, we divide this embed_dim by number_of_heads, so if embed_dim is 10, and number_of_heads is 2, then first head would use first 5 numbers, second head would use next 5 numbers.</p>
<p>where, Q is word which we want to get embedding for, K are all nearby words, V also represents all nearby words.</p>
<p>what is mistake in this explanation?<br>
and why do we use 3 * embed_dim?</p>
Change layer size

username: jojono
cooked: <p>I try to change the size of a linear model by adding one new neuron after few epochs.</p>
<pre><code class="lang-auto">net = torch.nn.Linear(4, 4)
optimizer = torch.optim.SGD(net.parameters(), lr=0.01)
#training here ...
net.weight = torch.nn.Parameter(torch.cat((backSave.weight, torch.randn(1, 4)), 0))
net.bias = torch.nn.Parameter(torch.cat((backSave.bias, torch.randn(1)), 0))
#training here again but the weights do not change anymore
</code></pre>
<p>However, when I then train the model again after updating the size, the model weights no longer change. The model is not trained anymore.<br>
I found a solution, I have to recreate the optimizer after the modification of the size</p>
<pre><code class="lang-auto">optimizer = torch.optim.SGD(net.parameters(), lr=0.01)
</code></pre>
<p>But I do not like this solution, is there another solution without defining again the optimizer<br>
Thank you</p>

username: Ditlev_Jorgensen
cooked: <p>I dont think you can change the size of your model during training. The optimizer is set to update a specific set of parameters. When you add new parameters, the optimizer dont know these exists, and thus needs to be reinitialized with the newly added parameters in order to work again.</p>

username: jojono
cooked: <p>I understand that the problem is that I redefine the parameter so I have to register it again to the optimizer.<br>
Is it possible to resize a layer without redefining its parameters ? I tried to use resize_ but it did not work because the parameter reque.</p>

username: Ditlev_Jorgensen
cooked: <p>I’ve found another guy who does exactly what you want to do (I think). And he also have to re-initialize the optimizer every time. Remember to resize bias as well not only weights.</p>
<p>He is resizing weights during training: <a href="https://discuss.pytorch.org/t/dynamically-expandable-networks-resizing-weights-during-training/19218">resizing weights during training</a></p>
<p>However, as seen in the comments, you cannot only resize_ because you have to copy and paste the old weights over. Its computationally heavy but should work in your case as well. Hope it helps!</p>
Can't find the source of a “expected…cuda…but got…cpu” error

username: ResidentMario
cooked: <p>I’m struggling to find the source of a RuntimeError stating that a certain input that’s supposed to be on GPU is actually on CPU. I know this is usually a “you forgot to add a .cuda()” somewhere error, but I can’t seem to find what I’m missing…could use a code review from a practitioner more experienced with PyTorch.</p>
<p><a href="https://stackoverflow.com/questions/58581464/pytorch-embedding-layer-raises-expected-cuda-but-got-cpu-error" rel="nofollow noopener">Here’s a StackOverflow question with more details, include the model code</a>.</p>

username: ResidentMario
cooked: <p>Solution has been posted:</p>
<blockquote>
<p>Pytorch requires your to do  <code>self.module_name = module</code>  for things to work correctly. It’s okay to keep them in a list. Just do something like  <code>setattr(self, 'emb_{}'.format(i), emb)</code>  for each step in that loop.</p>
</blockquote>
Getting same losses every epoch when Normalization is applied

username: Olament
cooked: <p>Hi, I am working on a depth estimation project using NYU Depth V2 Datasets. The depth image in the datasets is 16 bit so i convert it to tensor and normalize it with following code</p>
<pre><code class="lang-auto"># transformation
        comm_trans = transforms.Compose([
            transforms.Resize((240, 320)),
            transforms.CenterCrop((228, 304)),
            transforms.RandomHorizontalFlip()
        ])
        image_trans = transforms.Compose([
            transforms.ToTensor(),
            transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
        ])
        depth_trans = transforms.Compose([
            transforms.Resize((64, 80)),
            transforms.ToTensor(),
            transforms.Lambda(lambda x: x.float()),
            transforms.Lambda(lambda x: torch.div(x, 65535.0)),
            transforms.Normalize((0.5, ), (0.5, ))
        ])
        image = image_trans(comm_trans(image))
        depth = depth_trans(comm_trans(depth))
</code></pre>
<p>However, when training the network, i found that the loss of each batch is the same after one epoch. In other words, my network is not learning for some reasons.</p>
<blockquote>
<p>Epoch: [1/2] Step [20/885] Loss: 0.8645<br>
Epoch: [1/2] Step [40/885] Loss: 0.6820<br>
Epoch: [1/2] Step [60/885] Loss: 0.4783<br>
…<br>
Epoch: [2/2] Step [20/885] Loss: 0.9024<br>
Epoch: [2/2] Step [40/885] Loss: 0.6820<br>
Epoch: [2/2] Step [60/885] Loss: 0.4783<br>
…</p>
</blockquote>
<p>Then, i remove the scaling and normalization for depth images and everything works after that.</p>
<pre><code class="lang-auto">transforms.Lambda(lambda x: torch.div(x, 65535.0))
transforms.Normalize((0.5, ), (0.5, ))
</code></pre>
<p>So i am wondering what cause the problem and how do i fix it.</p>
<p>This is the hyperparameters and codes for training if that helps.</p>
<pre><code class="lang-auto"># hyperparameter
batch_size = 32
learning_rate = 0.001
total_epoch = 2
report_rate = 20

# optimizer and loss function
criterion = losses.BerHuLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

for epoch in range(total_epoch):
    running_loss = 0.0
    epoch_loss = 0.0
    for i, (image, depth) in enumerate(loader):
        image = image.to(device)
        depth = depth.to(device)

        # forward pass
        outputs = model(image)
        loss = criterion(outputs, depth)

        # Backward and optimize
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        # Calculate loss
        running_loss += loss.item()
        epoch_loss += running_loss
</code></pre>
<p>I did not post my model architecture due to space limitation. You can take a look at it here: <a href="https://github.com/Olament/DepthNet" class="inline-onebox" rel="noopener nofollow ugc">GitHub - Olament/DepthNet: Monocular depth estimation with CNN</a></p>
Network behaves differently after storing and loading

username: koelscha
cooked: <p>Hi,<br>
I stumbled across a problem recently, when I started to train a network, stored a checkpoint and after loading it again to resume the training, I got a huge spike in the loss. So, I investigated a little bit to figure out what is going on. After debugging, I found out, the network and optimizer are loading correctly.</p>
<p>The problem seems to be somehow connected to the custom Loss function, but I cannot find out what the problem is, as it looks fine to me and training actually works with it.</p>
<p>I created a minimal example to show the problem and created a gist for it: <a href="https://gist.github.com/koelscha/172505d7fe3c17b3db84282e6bb5caeb" rel="nofollow noopener">https://gist.github.com/koelscha/172505d7fe3c17b3db84282e6bb5caeb</a></p>
<p>So, if I set use_weights=False in the loss function, it works as expected. If I set use_weights=True, the weird behavior from above happens - this even happens if the weights are all 1 <img src="https://discuss.pytorch.org/images/emoji/apple/open_mouth.png?v=9" title=":open_mouth:" class="emoji" alt=":open_mouth:"></p>
<p>I am using pytorch version 1.3.0, an NVidia RTX 2080 Ti and CUDA 10.1</p>
<p>Thanks for your help advance,<br>
koelscha</p>

username: koelscha
cooked: <p>I think I found out, what the problem was. However, it’s incredible that this could happen without even a warning being printed.<br>
It seems, pytorch has a problem when multiplying two slices of tensors with different datatype. If I call .contiguous() on the tensors before multiplying, everything works fine. Minimal example: <a href="https://gist.github.com/koelscha/4dc9255a83e39b36b1cf20afa3c6fa74" rel="nofollow noopener">https://gist.github.com/koelscha/4dc9255a83e39b36b1cf20afa3c6fa74</a></p>
<p>I would expect to at least see a warning that this causes problems, since it works fine in numpy</p>
Displaying MNIST images

username: ati
cooked: <p>Hi<br>
I have the below code but i don’t know where is wrong that i get error</p>
<pre><code class="lang-auto">trainset = torchvision.datasets.MNIST(root ='./data',
                                        download=True,
                                        transform=transforms.Compose(
        [transforms.ToTensor(), transforms.Lambda(lambda x: x * 64)]
    ))
x= trainset[5] 
plt.imshow(x, cmap='gray')
</code></pre>
<p>here is the error:</p>
<pre><code class="lang-auto">ValueError: only one element tensors can be converted to Python scalars
</code></pre>

username: Dipam_Vasani
cooked: <p>What you are loading is the train_loader.  You can get a batch of images from it using</p>
<pre><code class="lang-python">images, labels = next(iter(train_loader))
</code></pre>
<p>The shapes of these are as follows</p>
<pre><code class="lang-python">images.shape
&gt; (torch.Size([128, 1, 28, 28]))
</code></pre>
<p>For plotting you need to use the last 2 dimensions (width and height)</p>
<pre><code class="lang-python">plt.imshow(images[0].reshape(28,28), cmap="gray")
</code></pre>

username: ati
cooked: <p>thank you …it worked for me<br>
just one question:<br>
in here:</p>
<pre><code class="lang-auto">(torch.Size([128, 1, 28, 28]))
</code></pre>
<p>what is 128 …is that batch_size?</p>

username: Dipam_Vasani
cooked: <p>Yes, it is the batch size.</p>
BCEWithLogitsLoss and model accuracy calculation

username: mese79
cooked: <p>Hi<br>
I have a simple binary classifier model, but I didn’t use <code>Sigmoid</code> at the end so I’ve trained my model with <code>BCEWithLogitsLoss</code> criterion.<br>
My targets(true labels) is created based on a value like <code>if value &gt;= 0</code> then target is <code>1</code> else <code>0</code> .<br>
The question is in testing what should I set as threshold? <code>0.5</code> or <code>0.0</code> ?</p>

username: KFrank
cooked: <p>Hi Mehdi!</p>
<aside class="quote no-group" data-username="mese79" data-post="1" data-topic="59293" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/ee59a6/48.png" class="avatar"> mese79:</div>
<blockquote>
<p>I have a simple binary classifier model, but I didn’t use <code>Sigmoid</code> at the end so I’ve trained my model with <code>BCEWithLogitsLoss</code> criterion.<br>
My targets(true labels) is created based on a value like <code>if value &gt;= 0</code> then target is <code>1</code> else <code>0</code> .<br>
The question is in testing what should I set as threshold? <code>0.5</code> or <code>0.0</code> ?</p>
</blockquote>
</aside>
<p>You should use 0.0 as your threshold for class “1”.</p>
<p>Here are the details:</p>
<p>A <em>logit</em> is a sort of score that runs from -infinity to +infinity.<br>
When you run it through the sigmoid function, you get a<br>
probability that runs from 0 to 1. (<code>BCEWithLogitsLoss</code> has,<br>
in effect, a sigmoid function inside of it.)  We interpret this<br>
probability as being the probability of class “1”.   So we (usually)<br>
convert such a probability to a yes-no prediction by saying if<br>
the probability of being class “1” is greater than 1/2, then we<br>
predict class “1” (and if it is less that 1/2, we predict class “0”).</p>
<p>The sigmoid function maps logits less than 0 to probabilities<br>
less than 1/2 (and logits greater than 0 to probabilities greater<br>
than 1/2, and sigmoid (0) = 1/2).</p>
<p>So, if your probability threshold is 1/2 (as it typically is), then<br>
your logit threshold is 0.</p>
<p>To recap:  As you’ve described it, the output of your model is a<br>
logit (because that is what you’ve trained your model to output),<br>
so to convert the output of your model to a yes-no prediction, use<br>
a threshold of 0.</p>
<p>Best.</p>
<p>K. Frank</p>

username: mese79
cooked: <p>Thanks for your thorough answer!</p>
Where is the Dataset and model file

username: Aspirinkb
cooked: <p>I’m trying to run the codes in <a href="https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html#experimental-static-quantization-with-eager-mode-in-pytorch" rel="nofollow noopener">(EXPERIMENTAL) STATIC QUANTIZATION WITH EAGER MODE IN PYTORCH</a>, but there are no dataset and model files available, such as <strong>imagenet_1k, mobilenet_quantization.pth</strong> and so on.<br>
So anyone can provide the address of the necessary files and dataset in this tutorial?</p>

username: bintonto
cooked: <p>Also, static quantification of google colab is not available</p>

username: jerryzh168
cooked: <p><a class="mention" href="/u/raghuramank100">@raghuramank100</a> can you take a look?</p>

username: Aspirinkb
cooked: <p><a class="mention" href="/u/raghuramank100">@raghuramank100</a> Need your help.</p>

username: raghuramank100
cooked: <p>Hi frank,  to run the tutorial, you will need to download imagenet dataset yourself, as we cannot upload imagenet images. The dataset can be obtained by using: <a href="https://pytorch.org/docs/stable/_modules/torchvision/datasets/imagenet.html#ImageNet" rel="nofollow noopener">https://pytorch.org/docs/stable/_modules/torchvision/datasets/imagenet.html#ImageNet</a></p>
<p>For the tutorial, you only need the floating point models as the tutorial walks through the process of creating a quantized model from a floating point model. The floating point mobilenet model can be found at: <a href="https://github.com/pytorch/vision/blob/master/torchvision/models/mobilenet.py#L9" rel="nofollow noopener">https://github.com/pytorch/vision/blob/master/torchvision/models/mobilenet.py#L9</a>.</p>
<p>We will add  more documentation to the tutorial so that these steps are clearer.</p>

username: Aspirinkb
cooked: <p>Thanks for your replay. I find the download urls are in the Makefile in github pytorch/tutorial.</p>
Max-Margin Loss Drops to Zero, but Metric still very low

username: Rafael_R
cooked: <p>Hi,</p>
<p>I am trying to train a shared embedding model using max-margin loss tested on an external retrieval metric.</p>
<p>The loss drops to zero just after one iteration!! and the metric is still very low.</p>
<p>I tried different margins, the same issue is there.</p>
<p>Any special tricks required to train under max-margin loss?</p>
Extract Patches and Return in __getitem__()

username: Abheesht_Sharma
cooked: <p>I have created n patches (say) of an image. How do I return one patch at a time in <strong>getitem</strong>() of my Image_Dataset class?</p>

username: ptrblck
cooked: <p>If you’ve created the patches in <code>__getitem__</code>, you could return all of them and create another loop over these patches inside your <code>DataLoader</code> loop, if necessary.<br>
Alternatively, if you’ve created the patches before and store them e.g. in a list, you could simply return a single patch by indexing the container (list).</p>

username: Abheesht_Sharma
cooked: <p>I returned the patches I created as a list in <code>__getitem__()</code> and accessed them individually in my training loop. Thank you for helping <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Error printing trainable/frozen parameters

username: Virilo_Tejedor_Aguil
cooked: <p>I created this function to print the model, and mark the trainable/frozen modules:</p>
<pre><code class="lang-auto">def get_model_desc(model):
    network_desc=""
    for name, p in model.state_dict().items():
        #p.requires_grad=True
        frozen= " ·t· " if p.requires_grad else " [F] "
        network_desc_spaces=" "*(55-len(name))
        network_desc+=f"{name} {network_desc_spaces} {frozen} {list(p.size())}\n"
    return network_desc
print(get_model_desc(model))
</code></pre>
<p>It shows al the parameters as frozen, but they musn’t be frozen.</p>
<p>I must be missing something, because there are modules whose parameters I have explicitly assigned as  requires_grad=True</p>
<p>Also, I uncommented without success the line p.requires_grad=True</p>

username: ptrblck
cooked: <p>The <code>state_dict</code> will store all parameters as tensors, which do not require gradients.<br>
Have a look at this small example:</p>
<pre><code class="lang-python">model = models.resnet18()

for p in model.parameters():
    print(p.requires_grad)

for _, p in model.state_dict().items():
    print(p.requires_grad)
</code></pre>
<p>If you want to check the parameters of a model, use <code>model.parameters()</code> instead.</p>

username: Virilo_Tejedor_Aguil
cooked: <p>thanks a lot <a class="mention" href="/u/ptrblck">@ptrblck</a> !</p>
Autograd.detect_anomaly() in CNN

username: vihas_makwana
cooked: <p>Hi, I am using autograd.detect_anomaly() on my CNN and getting following output and I didn’t get it.</p>
<p><a href="https://2gh8ivti7ma-b5278c2458b95a70-0-colab.googleusercontent.com/outputframe.html?vrz=colab-20191024-080000-RC00_276488290#" rel="nofollow noopener">/usr/local/lib/python3.6/dist-packages/torch/autograd/<strong>init</strong>.py</a> in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables) 97 Variable._execution_engine.run_backward( 98 tensors, grad_tensors, retain_graph, create_graph, —&gt; 99 allow_unreachable=True) # allow_unreachable flag 100 101</p>
<p>RuntimeError: Function ‘PowBackward0’ returned nan values in its 0th output.</p>

username: albanD
cooked: <p>You get this output because one of the <code>pow</code> (power) operation returned nan values. Since you usually don’t want nan value, this is most likely a problem somewhere.</p>