akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Mutual / Lateral inhibition in a single layer

username: KarimHabashy
cooked: <p>Hi,</p>
<p>Is there a away to apply mutual / lateral inhibition, in a linear layer, where there is only one winner (value near 1) and the rest are inhibited (near 0) in a differential-able way ?</p>
<p>Thanks</p>

username: KFrank
cooked: <p>Hi Karim!</p>
<aside class="quote no-group" data-username="KarimHabashy" data-post="1" data-topic="63342" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ce73a5/48.png" class="avatar"> KarimHabashy:</div>
<blockquote>
<p>Is there a away to apply mutual / lateral inhibition, in a linear layer, where there is only one winner (value near 1) and the rest are inhibited (near 0) in a differential-able way ?</p>
</blockquote>
</aside>
<p>If you want to have <code>n</code> mutually-inhibitory values, <code>x_i</code>, you can<br>
add an “inhibition” loss term to your overall loss function.</p>
<p>The idea is that you want your loss to be small (say, zero) when<br>
you are at (or near) the <code>n</code> special points you prefer.  (These <code>n</code><br>
points are where one of the <code>n</code> <code>x_i</code> is 1, and the others are zero.)</p>
<p>So we take the (squared) distance of your actual <code>x_i</code> from each<br>
of those preferred points and multiply those <code>n</code> distances together.<br>
Because you multiply them all together, being close to any of the<br>
preferred points makes your loss small, and being exactly at one<br>
of those points makes your loss exactly zero.</p>
<p>Here’s a formula for the above words:</p>
<pre><code class="lang-plaintext">inh = Prod_{i} ( (x_i - 1)^2 + Sum_{j != i} (x_j)_^2 )
</code></pre>
<p>We built this inhibition term, <code>inh</code>, out of differentiable pieces, and<br>
combined them together in a differentiable way, so <code>inh</code> is fully<br>
differentiable.</p>
<p>(I assume that you want the output of your linear layer to display this<br>
inhibition, rather than, say, the parameters, but this scheme doesn’t<br>
depend on where the <code>x_i</code> come from.)</p>
<p>Good luck.</p>
<p>K. Frank</p>
Learning rate scheduling in PyTorch

username: Shobhit_Verma
cooked: <p>Hello<br>
I was trying to train a model and adjust the learning rate after a few epochs to see the effects on training.<br>
Specifically, I’d like to adjust the learning rate when the loss function ‘plateus’<br>
Is there a way to do this in PyTorch?</p>

username: mmisiur
cooked: <p>Read the docs: <a href="https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate" rel="nofollow noopener">https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate</a></p>
<p>There are many ways to do this, including plateau.</p>

username: Shobhit_Verma
cooked: <p>Thank you! I somehow couldn’t find this, my bad</p>
What's the standard of Conv2d input?

username: ziming_zhong
cooked: <p>‘’’<br>
import torchvision<br>
import numpy as np<br>
from PIL import Image<br>
import torch<br>
model = torchvision.models.resnet50(pretrained=True)<br>
k = Image.open(’/p300/data/images/000b7d55b6184b08.png’)<br>
k = np.array(k)<br>
k = np.expand_dims(k, axis=0)<br>
k = torch.Tensor(k)<br>
result = model(k)<br>
‘’’<br>
k is an image whose size is [299,299,3]</p>
<p>And the error is RuntimeError: Given groups=1, weight of size 64 3 7 7, expected input[1, 299, 299, 3] to have 3 channels, but got 299 channels instead.</p>
<p>I already have change the k.shape from [299,299,3] to [1,299,299,3].<br>
How the conv2d recognized the input channels?</p>

username: Eta_C
cooked: <p>It should be (N, C, H, W), not (N, H, W, C)</p>

username: ziming_zhong
cooked: <p>After loading images with Image.open, I just get the result in (C, H, W, C). How can I change it to (N, C, H, W) in a simple way? Because I find if I use the</p>
<blockquote>
<p>trans = transform.Compose([ transforms.Resize((299,299)),<br>
transforms.ToTensor(),<br>
transforms.Normalize(mean=[0.485, 0.456, 0.406],<br>
std=[0.229, 0.224, 0.225])</p>
</blockquote>
<blockquote>
<p>k = trans(k)</p>
</blockquote>
<p>the codes will work. So is there any better way to achieve it?</p>
Using a different box predictor with FasterRCNN

username: ssetu
cooked: <p>Hi,<br>
I am trying to solve an object detection problem using FasterRCNN, where I have <em>N</em> detection scenarios. I am getting good results with transfer learning just the last layer (box predictor), but I end up having <em>N</em> models, one for each scenario. Consequently, it takes up a lot of storage space and RAM if I want all the models to be loaded simultaneously. Now, only the box predictor (last layer) is different for each network, so I want to pass my inputs in a batch, get the output from forward pass till only the last layer, and then pass the individual outputs to <em>N</em> different box predictors. What’s the best strategy to implement this? I can only think of swapping the last layer before running on individual inputs, which is too slow.</p>
<p>Thanks</p>

username: spanev
cooked: <p>Hi <a class="mention" href="/u/ssetu">@ssetu</a>,</p>
<p>I think the best way to achieve that would be to create a Module inheriting from <code>FasterRCNN</code> (minus the <code>box_predictor</code>) with a list of your fine-tuned <code>box_predictor</code>s, define a custom <code>forward</code> with an extra “detection scenario” argument where you would switch on the right <code>box_predictor</code>, depending on that extra argument.</p>

username: ssetu
cooked: <p>Thank you <a class="mention" href="/u/spanev">@spanev</a>. I will try this out with clues from <a href="https://discuss.pytorch.org/t/add-multiple-fc-layers-in-parallel/19008/8" class="inline-onebox">Add multiple FC layers in parallel</a> and post an update soon.</p>
How to save memory by doing op like x = conv(x) while inferencing

username: 111197
cooked: <p>Hi, I am trying to run my code in “torch.no_grad” and “eval” model. I want to save memory via operations like x = conv(x). But while I running the following code:</p>
<pre><code class="lang-auto">x = self.conv1(x)
x = self.conv2(x)
</code></pre>
<p>I found it would  consume 3 times memory as much as x, it seems that the first x is not be replaced. But Ideally, it could be reduced to 2 times (since conv operations need extra memory). I have finished the training process, how could I free the memory of the first x after ops like x = conv(x)?<br>
thx!</p>

username: ptrblck
cooked: <p>I doubt you can save more memory than is already saved by wrapping the code in a <code>with torch.no_grad()</code> block. The output of <code>conv1</code> has to be allocated at one place during the forward pass.<br>
How many output channels are you using?</p>

username: 111197
cooked: <p>Sorry, I didn’t make myself clear. The code is like this:</p>
<pre><code class="lang-auto">def forward(self, x):
    x = conv1(x)
    x = conv2(x)
  
</code></pre>
<p>After calculating the conv1, the memory of the very first x should be free since it is useless anynore. But pytorch dose not free it. Is any solution to free the input x after the conv1.</p>

username: ptrblck
cooked: <p>If <code>x</code> is not needed anymore, PyTorch will free it and reuse the memory.<br>
In case you doubt it, you could try to assign the result of <code>conv1</code> to another variable and delete <code>x</code> manually.</p>

username: 111197
cooked: <p>Actually I test it based on a resnet architecture.<br>
The whole code would like this:</p>
<pre><code class="lang-auto">Class BasicBlock(nn.Module):
    def __init__():
        super(BasicBlock, self).__init__()
        self.conv1 = nn.Conv2d(64, 64, 3)
        self.conv2 = nn.Conv2d(64, 64, 3)
    def forward(self, x):
        out = self.conv1(x)
        out = self.conv2(x)
    return x
class WholeModel(nn.Module):
    def __init__():
        super(WholeModel, self).__init__()
        self.block1 = BasicBlock()
        self.block2 = BasicBlock()
    def forward(self, x):
        out = self.block1(x)
        out = self.block2(out)
</code></pre>
<p>While runing the “out = self.block1(x)”, it saves x. And after the first conv in block1: “out = self.conv1(x)”. x is useless anymore. But the input x will not be freed until “out = self.block1(x)” is finished. I think it could be freed earlier since it is useless anymore after the conv1 in block1 is finished. I tried to “del x” after the “out = self.conv1(x)”, but it is obviously useless. Do you have any ideas about how to free the  memory just after the “out  = self.conv1(x)”? Thank youx</p>
Can BatchNorm2d weights be negative?

username: hashbrown
cooked: <p>When I train an network, I see that the BatchNorm2d weights are becoming negative. Is this possible? Any way to prevent this?</p>

username: ptrblck
cooked: <p>In the default setup batchnorm layers use affine parameters (<code>weight</code> and <code>bias</code>) as well as the running estimates. Neither of those tensors is bounded in any way.<br>
If you want to enforce a certain range you might e.g. clip the weight after an update.</p>
How to multiply tensor on mobile

username: Kiki_Rizki_Arpiandi
cooked: <p>I need to multiply tensor for calculating priors/anchorbox for my object detection model on android does Java has tensor multipication operation yet?</p>

username: David_Reiss
cooked: <p>No, and we have no plans to add it.  Please update your TorchScript model to do any tensor operations you need so that the Java code can simply provide raw input and consume raw output.</p>
Where is compiled libtorch location?

username: Kiki_Rizki_Arpiandi
cooked: <p>I compiled libtorch for ARM following this guide <a href="https://github.com/pytorch/pytorch/blob/master/docs/libtorch.rst" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/docs/libtorch.rst</a>. But I did not find torch/lib/tmp_install  directory. Where is the directory?</p>

username: Kiki_Rizki_Arpiandi
cooked: <p>Finally I found it. it is located at</p>
<blockquote>
<p>pytorch/torch</p>
</blockquote>
Onnx2trt generates trt model, how to load in tensorrt

username: jinfagang
cooked: <p>I know this is not a pytorch issue, but since onnx model would gain a huge performance if using tensorrt for inference, must many people have tried this.</p>
<p>I want ask I have generate a mobilenetv2.trt model with onnx2trt tool, how do I load it in tensorrt?</p>
<p>Have anyone could provide a basic inference example of this?</p>
<p>Most usage I got is loading model directly from onnx and parse it with NvOnnxParser, since we generate a trt model, I think this step is unessary…</p>
<p>I will give him a bitcion if anyone solve  my question</p>

username: Naman_Bhayani
cooked: <p>Hii, I have ran Object Detectors using TensorRt. You may contact me at <a href="mailto:namanbhayani.bitspilani@gmail.com">namanbhayani.bitspilani@gmail.com</a> if you need help.</p>

username: jinfagang
cooked: <p><a class="mention" href="/u/naman_bhayani">@Naman_Bhayani</a> Which model are u using?</p>
<p>I try to send you an email, helpfully you can guide me somewhere</p>

username: Naman_Bhayani
cooked: <p>Hey <a class="mention" href="/u/jinfagang">@jinfagang</a></p>
<p>I have created a repo for the same. Can you check if this help?</p>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://github.com/Rapternmn/PyTorch-Onnx-Tensorrt" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars2.githubusercontent.com/u/8393438?s=400&amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/Rapternmn/PyTorch-Onnx-Tensorrt" target="_blank" rel="nofollow noopener">Rapternmn/PyTorch-Onnx-Tensorrt</a></h3>

<p>A set of tool which would make your life easier with Tensorrt and Onnxruntime. This Repo is designed for YoloV3 - Rapternmn/PyTorch-Onnx-Tensorrt</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Mistake on pytorch documentation

username: dato_nefaridze
cooked: <p>here is what nn.Linear do but i am confused.i think thre should be A^T*x and not vice versa<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/5/0583c6086cdc9c776c261b99e4539fa831f4ecf8.png" data-download-href="https://discuss.pytorch.org/uploads/default/0583c6086cdc9c776c261b99e4539fa831f4ecf8" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/5/0583c6086cdc9c776c261b99e4539fa831f4ecf8_2_690x387.png" alt="image" data-base62-sha1="MMI4L3LE5mnWQZdip3nfmZZmdi" width="690" height="387" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/5/0583c6086cdc9c776c261b99e4539fa831f4ecf8_2_690x387.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/5/0583c6086cdc9c776c261b99e4539fa831f4ecf8_2_1035x580.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/5/0583c6086cdc9c776c261b99e4539fa831f4ecf8.png 2x" data-dominant-color="F6F5F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1366×768 93.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>The documentation refers to the implemented method as seen <a href="https://github.com/pytorch/pytorch/blob/6848f9abb82edc4e03e34b2c45a6c559a45a6c7c/torch/nn/functional.py#L1375">here</a>.</p>
<p>Your suggestion won’t work with plain <code>matmul</code>, as <code>x</code> has the batch dimension in dim0.</p>
How should I weight my embeddings before aggregating them?

username: eugene
cooked: <p>I’m currently building an implementation of node2vec where each node can have certain metadata attributes, such as the example below:</p>
<ul>
<li>node: x</li>
<li>category: abc</li>
<li>brand: xyz</li>
</ul>
<p>To embed a node, I’ve trained embeddings for each feature (i.e., node, category, brand, which all have different dimensions). Then, I do torch.mean, such as the example below:</p>
<pre><code class="lang-auto">emb_nodes = []
for i in range(nodes.shape[1]):
    emb_nodes.append(self.embeddings[i](nodes[:, i]))
emb_agg = torch.mean(torch.stack(emb_nodes), axis=0)
</code></pre>
<p>However, this doesn’t work in practice and I think I need to weight how these features (i.e., node, category, brand) are aggregated for each node id.</p>
<p>I’ve looked into embeddingBag but it only allows the embedding dimensions for all embeddings to be the same–in my case, they are all varying (cardinality of product is much larger than category and brand).</p>
<p>Anyone have any suggestions and sample code on how I could train such a weighting layer efficiently? Should I put the weights in another embedding layer?</p>
Conv2d and padding_mode: circular vs reflect

username: brookisme
cooked: <p>Hi All -</p>
<p>I was looking into padding-mode for <code>nn.Conv2d</code>.    In the <a href="https://pytorch.org/docs/stable/nn.html#conv2d" rel="noopener nofollow ugc">docs</a> it doesn’t describe the options but in the <a href="https://pytorch.org/docs/stable/_modules/torch/nn/modules/conv.html#Conv2d" rel="noopener nofollow ugc">source code</a> its says</p>
<blockquote>
<p>padding_mode (string, optional). Accepted values <code>zeros</code> and <code>circular</code></p>
</blockquote>
<p>Digging deeper, padding mode calls <code>F.pad()</code>, and <code>F.pad()</code> has the following options for mode</p>
<blockquote>
<p>mode: <code>'constant'</code>, <code>'reflect'</code>, <code>'replicate'</code> or <code>'circular'</code></p>
</blockquote>
<p>I was originally looking to use “reflect”. Questions:</p>
<ul>
<li>is reflect implemented but not in the docs (for <code>nn.Conv2d</code>)?</li>
<li>if not implemented, any particular reason?</li>
<li>i’ve never used “circular” padding before:
<ul>
<li>what is it?</li>
<li>when is it good to use?</li>
<li>is there general advice on if its best to use zero-padding/circular/reflect?</li>
</ul>
</li>
<li>what is “replicate” (does it just extend the borders out with the edge values)</li>
</ul>
<p>Thanks!</p>

username: Marvek
cooked: <p><code>padding_mode</code> of Con2d accept values <code>zeros</code>  and  <code>circular</code> only.</p>
<p>Looking the code of Conv2d:</p>
<pre><code class="lang-auto">class Conv2d(_ConvNd):
    def __init__(self, in_channels, out_channels, kernel_size, stride=1,
                 padding=0, dilation=1, groups=1,
                 bias=True, padding_mode='zeros'):
        kernel_size = _pair(kernel_size)
        stride = _pair(stride)
        padding = _pair(padding)
        dilation = _pair(dilation)
        super(Conv2d, self).__init__(
            in_channels, out_channels, kernel_size, stride, padding, dilation,
            False, _pair(0), groups, bias, padding_mode)

    def conv2d_forward(self, input, weight):
        if self.padding_mode == 'circular':
            expanded_padding = ((self.padding[1] + 1) // 2, self.padding[1] // 2,
                                (self.padding[0] + 1) // 2, self.padding[0] // 2)
            return F.conv2d(F.pad(input, expanded_padding, mode='circular'),
                            weight, self.bias, self.stride,
                            _pair(0), self.dilation, self.groups)
        return F.conv2d(input, weight, self.bias, self.stride,
                        self.padding, self.dilation, self.groups)

    def forward(self, input):
        return self.conv2d_forward(input, self.weight)
</code></pre>
<p>if <code>padding_mode</code> is not <code>circular</code>, the code</p>
<pre><code class="lang-auto">return F.conv2d(input, weight, self.bias, self.stride,
                        self.padding, self.dilation, self.groups)
</code></pre>
<p>will always be executed.</p>
How to add nn.Embedding to model's parameters?

username: Audrey
cooked: <p>Hello! I want to know how to add nn.Embedding type variables to model’s parameters. My model is defined as below:</p>
<pre><code class="lang-auto">class Model(nn.Module):
    def __init__(self, a, b, hops):
        self.W1 = torch.nn.Embedding(a, b)
        self.W2 = torch.nn.Embedding(a, b)
        self.R_list = []
        for i in range(hops):
            R_list.append(torch.nn.Embedding(a, b))
</code></pre>
<p>When I check <code>model.parameters()</code>, only <code>W1</code> and <code>W2</code>. are outputed, which means that <code>R</code> are not in model.paremeters. So I want to know how to add them. Thanks!!</p>
<hr>
<p>I know <code>torch.nn.Parameter()</code> can put Tensor type in the model’s parameter.</p>

username: vainaijr
cooked: <aside class="quote no-group quote-modified" data-username="Audrey" data-post="1" data-topic="63387">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/audrey/48/18516_2.png" class="avatar"> Audrey:</div>
<blockquote>
<p>class Model(nn.Module): def <strong>init</strong>(self, a, b, hops): self.W1 = torch.nn.Embedding(a, b) self.W2 = torch.nn.Embedding(a, b) self.R_list = <span class="chcklst-box fa fa-square-o fa-fw"></span> for i in range(hops): R_list.append(torch.nn.Embedding(a, b))</p>
</blockquote>
</aside>
<p>do you mean something like this,</p>
<pre><code class="lang-auto">class Model(nn.Module):
    def __init__(self, a, b, hops):
        super().__init__()
        self.W1 = torch.nn.Embedding(a, b)
        self.W2 = torch.nn.Embedding(a, b)
        self.R_list = nn.ModuleList()
        for i in range(hops):
            self.R_list.append(torch.nn.Embedding(a, b))
</code></pre>

username: Audrey
cooked: <p>Yeah, <code>nn.ModuleList()</code> works for me! Thanks. <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji" alt=":grinning:"></p>
How to define the __len__ method for PyTorch Dataloader when I have separate length datasets?

username: seankala
cooked: <p>I’m currently loading in my data with one single dataset class. Within the dataset, I split the train, test, and validation data separately. For example:</p>
<pre><code class="lang-auto">class Data():
    def __init__(self):
        self.load()

    def load(self):
        with open(file=file_name, mode='r') as f:
            self.data = f.readlines()

        self.train = self.data[:checkpoint]
        self.valid = self.data[checkpoint:halfway]
        self.test = self.data[halfway:]
</code></pre>
<p>Many of the details have been omitted for the sake of readability. Basically, I read in one big dataset and make the splits manually.</p>
<p>My question is arising from how to override the <code>__len__</code> method when the lengths of my train, valid, and test data all differ?</p>
<p>The reason I want to do this is because I want to keep the split data in one single class, and I also want to create separate Dataloaders for each, and so something like:</p>
<pre><code class="lang-auto">def __len__(self):
    return len(self.train)
</code></pre>
<p>wouldn’t be appropriate for <code>self.test</code> and <code>self.valid</code>.</p>
<p>Perhaps I’m fundamentally misunderstanding the Dataloader, but how should I approach this issue? Thanks in advance.</p>

username: ptrblck
cooked: <p>I’m not sure there is a clean way of handling different subsets within a single <code>Dataset</code> class.<br>
If you want to handle the split yourself, I would rather create a custom function or class (if statefull) and return the corresponding dataset using e.g. <code>TensorDataset</code> or <code>Subset</code>.</p>
<p>E.g. this might be a usable code base:</p>
<pre><code class="lang-python">
class MyDatasetSplitter(object):
    def __init__(self, checkpoint, halfway):
        self.load()
        self.datasets = {}
        self.checkpoint = checkpoint
        self.halfway = halfway
        
    def load(self):    
        with open(file=file_name, mode='r') as f:
            self.data = f.readlines()

        self.datasets['train'] = TensorDataset(
            self.data[:self.checkpoint])
        self.dataset['valid'] = TensorDataset(
            self.data[self.checkpoint:self.halfway])
        self.datasets['test'] = TensorDataset(
            self.data[self.halfway:])
        
    def get_dataset(self, split):
        return self.datasets[split]
</code></pre>
<p>Would that work for your use case?<br>
In this way, each split is a custom <code>Dataset</code> and can be wrapped by a <code>DataLoader</code> separately.</p>

username: seankala
cooked: <p>Yes, this is also a method that would work. I could also just create separate dataset classes, but was wondering if there was a clean way to handle this issue. It seems not, unfortunately. <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>
Training using Dataloader; running time doesn't match

username: ElleryL
cooked: <p>Hi; I’m training a model on CeleberityA dataset as follows</p>
<pre><code class="lang-auto">transform = transforms.Compose([
        transforms.Resize((64, 64)),
        transforms.ToTensor(),
        transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),
    ])
    trainset = torchvision.datasets.CelebA(root='./data', split="train",
                                           download=True, transform=transform)
    train_loader = torch.utils.data.DataLoader(trainset, batch_size=args.batch_size,
                                              shuffle=True, num_workers=128)

    testset = torchvision.datasets.CelebA(root='./data', split="test",
                                          download=True, transform=transform)
    test_loader = torch.utils.data.DataLoader(testset, batch_size=args.batch_size,
                                             shuffle=False, num_workers=128)
</code></pre>
<p>Now my code looks like this</p>
<pre><code class="lang-auto">for epoch in range(total_epochs):
     Train()
</code></pre>
<p>where</p>
<pre><code class="lang-auto">def Train():
    model.train()

   epoch_start_time = time.time() # track how long time it take for this entire epoch
   total_time_used = 0
    for batch_idx, (data, _) in enumerate(train_loader):

        batch_start_time = time.time() # track how long time it take for one batch
        start_time = time.time()
        data = data.to(device) # device is a cuda
        optimizer.zero_grad()
        loss = compute_loss(data)
        loss.backward()
        optimizer.step()

        total_time_used += time.time() - batch_start_time # add the time spent for this batch

    print(time.time() - epoch_start_time)) # this gives the entire epoch's run time
    print(total_time_used) 
</code></pre>
<p>However, I’ve found that <code>total_time_used  !=  time.time() - epoch_start_time)</code> different in around 30s</p>
<p>I’m wondering why is this the case ?</p>

username: Joker
cooked: <p>Nope ,  not seeing  any   problem  in  your    code    , maybe  you   should  buy  a     better  cpu  ,better   GPU.       And      it  will  be  very  fast  .</p>

username: ElleryL
cooked: <p>Hi; I’ve checked the code with <code>time.time()</code> function and updated the post to indicate that there seems some time mismatched. Do you mind taking a look ? thanks</p>

username: ptrblck
cooked: <p>The first execution of the dataloader loop (<code>for data in loader</code>) won’t be recorded in <code>total_time_used</code>.<br>
This would correspond to the initial loading all a batch using all specified workers.</p>
Map high-level (graph) IR to low-level IR

username: soumyarooproy
cooked: <p>Hi,</p>
<p>I started looking into Glow compiler this week.</p>
<p>What is the recommended way to dump the dataflow graph for, say, the <a href="https://github.com/pytorch/glow/blob/master/docs/Example.md" rel="noopener nofollow ugc">LeNet example here</a> in dot format?</p>
<p>I see in <a href="https://github.com/pytorch/glow/blob/master/docs/IR.md" rel="noopener nofollow ugc">Glow IR documentation</a>:</p>
<blockquote>
<p>“The compiler has a debug method for dumping a graphical representation of the graph into a dotty file. The method is called ‘dumpDAG’. The images above were generated with this method. The textual representation of the graph is less informative and it looks like this…”</p>
</blockquote>
<p>However, I’m not sure I see a cmdline option to dump such a representation. Is that right? Do I need to hack it? I see that a <code>-dump-llvm-ir</code> option is available to <em>dump the LLVM-IR of the jitted code</em>.</p>
<p>My objective is to understand (visually) the mapping of the high-level constructs captured by the high-level IR to the linear algebra primitives with which the low-level IR is comprised.</p>
<p>-SR</p>

username: soumyarooproy
cooked: <p>Made some progress here. Documenting for posterity.</p>
<p>In the context of the LeNet example <a href="https://github.com/pytorch/glow/blob/master/docs/Example.md" rel="nofollow noopener">here</a>, described in C++, adding the following will dump the network DAG:</p>
<pre><code class="lang-auto">    F-&gt;dumpDAG("lenet.dot");
</code></pre>
<p>The dumped DAG is the one that’s shown in the link above. Run the following to generate the image:</p>
<pre><code class="lang-auto">dot -Tpng -O lenet.dot
</code></pre>
<p>To dump the low-level IR for the compilation unit — a function at a time as I understand (is that right?) — add the following:</p>
<pre><code class="lang-auto">    // Compile for inference
    EE.compile(CompilationMode::Infer, F);
    EE.getIR().dumpDAG("lenet_infer_ir.dot");
</code></pre>
<p>Alternately, to dump the training graph and the IR DAG for the training network, add the following:</p>
<pre><code class="lang-auto">    // Set up training of the network
    EE.getConfig().learningRate = 0.1;
    EE.getConfig().momentum = 0;
    EE.getConfig().batchSize = minibatchSize;

    Function *TF = glow::differentiate(F, EE.getConfig());
    // Compile for training
    EE.compile(CompilationMode::Train, TF);
    TF-&gt;dumpDAG("lenet_train.dot");
    EE.getIR().dumpDAG("lenet_train_ir.dot");
</code></pre>
<p>-SR</p>

username: qcolombet
cooked: <p>To add on the answer, if you run something that use the loader (Loader.cpp), you can use the command line option <code>-dumpGraphDAG=&lt;filename.dot&gt;</code> to dump the graph using the .dot format.</p>

username: Sharath_R
cooked: <p>Can we get the <strong>dumpDAG</strong> output for the python model too like in C++?</p>
Targets for multi-task semantic segmentation

username: joekid
cooked: <p>I appeal to your experience, I am newbie  in DL.</p>
<p>I like to test the semantic segmentation performance into a multitask, in which,  a task specializes on a single label. In this scenario, each task has its own loss, so each task requires its own targets.</p>
<p><strong>My main question is how to built the targets for each task ?</strong> I am working on <code>CamVid</code> dataset (on this, a target annotated image has 12 clases/labels ),  Should I post-process each annotated image (if yes, how it would be the class ‘0’ zero, for example)?</p>
<p>All suggestions are welcome.</p>
How to create a list of modulelists

username: orena1
cooked: <p>Hi,<br>
Is it ok to create a list of module list?<br>
If for example I want to have a few Conv1d in a layer and than another layer with different Conv1d, in each layer I need to do a different manipulation on the output depending on the layer,  what is the correct way to build this “list” of modulelists?</p>
<p>This way:</p>
<pre><code class="lang-auto">    class test(nn.Module):
        def __init__(...):
            self.modulelists = []
            for i in range(4):
                self.modulelists.append(nn.ModuleList([nn.Conv1d(10, 10, kernel_size=5) for _ in range(5)]))
</code></pre>
<p>or this way:</p>
<pre><code class="lang-auto">    class test(nn.Module):
        def __init__(...):
            self.modulelists = nn.ModuleList()
            for i in range(4):
                self.modulelists.append(nn.ModuleList([nn.Conv1d(10, 10, kernel_size=5) for _ in range(5)]))
</code></pre>
<p>Thanks</p>
How to print more than 4 digits after decimal?

username: vainaijr
cooked: <p>I get tensor as</p>
<pre><code class="lang-auto">tensor([[[0.1200],
         [0.0700]],

        [[0.0216],
         [0.0048]],

        [[0.0026],
         [0.0035]],

        [[0.0002],
         [0.0004]]])
</code></pre>
<p>how do I print after four digits, that is something like, 0.120001203, 0.070020133?</p>
<pre><code class="lang-auto">f'{x:.5f}'
</code></pre>
<p>throws error</p>
<pre><code class="lang-auto">TypeError: unsupported format string passed to Tensor.__format__
</code></pre>

username: ptrblck
cooked: <p>You could use <a href="https://pytorch.org/docs/stable/torch.html#torch.set_printoptions"><code>torch.set_printoptions</code></a> with the <code>precision</code> or <code>profile</code> argument to get more decimals.</p>

username: vainaijr
cooked: <p>thanks, it works now.</p>
Attention to SeqToOne model

username: VikasRajashekar
cooked: <p>Hello, I have a model that does sentiment analysis.<br>
I want to add attention to the model in order to improve accuracy.</p>
<p>In pytorch tutorial they have attention on decoder LSTM. But in my case the decoder is just an FC that decides the sentiment. How do I add attention to it?</p>
<p>Here is my network:<br>
import torch.nn as nn</p>
<pre><code class="lang-auto">class Intent_LSTM(nn.Module):
    """
    We are training the embedded layers along with LSTM for the sentiment analysis
    """

    def __init__(self, vocab_size, output_size, embedding_dim, hidden_dim, n_layers, drop_prob=0.5):
        """
        Settin up the parameters.
        """
        super(Intent_LSTM, self).__init__()

        self.output_size = output_size
        self.n_layers = n_layers
        self.hidden_dim = hidden_dim
        
        # embedding layer and LSTM layers 
        self.embedding = nn.Embedding(vocab_size, embedding_dim)
        self.lstm = nn.LSTM(embedding_dim, hidden_dim, n_layers, 
                            dropout=drop_prob, batch_first=True)
        
        # dropout layer to avoida over fitting
        self.dropout = nn.Dropout(0.5)
        
        # linear and sigmoid layers
        self.fc = nn.Linear(hidden_dim, output_size)
        

    def forward(self, x):
        """
        Perform a forward pass

        """
        batch_size = x.size(0)

        x = x.long()
        embeds = self.embedding(x)

        lstm_out, hidden = self.lstm(embeds)

    
        # stack up lstm outputs
        lstm_out = lstm_out.contiguous().view(-1, self.hidden_dim)


        out = self.dropout(lstm_out)
        out = self.fc(out)


        
        # reshape to be batch_size first
        out = out.view(batch_size, -1,3)
        #print("sig_out",sig_out.shape)
        out = out[:, -1,:] # get last batch of labels
        
        # return last sigmoid output and hidden state
        return out
</code></pre>
<p>Where exactly does the attention mechanism fit in this scenario? and what are inputs to it?</p>

username: vainaijr
cooked: <p>maybe you could go with nn.TransformerEncoder, it takes word embeddings as input, and output is learned representation, which could be fed to nn.Linear, and then nn.Softmax.<br>
or even check out <a href="http://github.com/huggingface/transformers" rel="nofollow noopener">github.com/huggingface/transformers</a></p>
What's mean 'shape '[10, 65536]' is invalid for input of size 1966080'

username: Foolish
cooked: <p>File “D:/360Downloads/VS/MyDataset.py”, line 233, in <br>
x=x.view(x.size(0),256*256)</p>
<p>RuntimeError: shape ‘[10, 65536]’ is invalid for input of size 1966080</p>

username: Eta_C
cooked: <p>The shape of your <code>x</code> is <code>(10, 3, 256, 256)</code>.<br>
use <code>x = x.view(x.size(0), -1)</code><br>
or <code>x = x.view(x.size(0), 3 * 256 * 256)</code></p>

username: Eta_C
cooked: <p>Boy, this is your second topic I responded to.<br>
You <strong>REALLY</strong> need to be more careful with your code…<img src="https://discuss.pytorch.org/images/emoji/apple/rofl.png?v=9" title=":rofl:" class="emoji" alt=":rofl:"></p>

username: Foolish
cooked: <p>all right,thans for your answer</p>
Channel autoencoder without AWGN

username: Filarius
cooked: <p>I’m working on problem to develop data encoding/modulation for transmission through channel what alters data. As example, you want to put data in lossy image like JPEG without hiding it (no steganograthy).<br>
In many works related to channel autoencoders I see AWGN (gauss noise) is used to represent noise of channel - in autoencoder there AWGN layer added between encoder and decoder.<br>
Its works good to make neural networks learn how to handle some random noise in latent representation, but in case of JPEG we will have data “shift” what will fully depends on input data rather than truly random noise, so its must be possible to use channel “non-randomess” to be learned by autoencoder.<br>
But I can’t find related examples to use.<br>
In my kind of this problem I take channel start/stop operations as very costy, and transmission delay as significant. In some cases you will need to send like 100 Mb  before you will be able to get copy of recieved data what can be used in training.</p>
Convert a numpy array to sparse tensor

username: zcajiayin
cooked: <p>Say I have a <code>numpy  array</code> like A:</p>
<pre><code class="lang-auto">[[1, 0, 0],
 [1, 0, 0],
 [0, 0, 1]]
</code></pre>
<p>How can I transfer A as a sparse tensor B?</p>

username: Eta_C
cooked: <p><a href="https://pytorch.org/docs/stable/sparse.html" rel="nofollow noopener">TORCH.SPARSE</a></p>

username: zcajiayin
cooked: <p>Thank you for replying. But the sparse tensor is in COO format which means I need to know coordinates and values to create one. But the situation here is that I want to get B from A directly. Is there a way for doing this?</p>

username: Eta_C
cooked: <p>torch.from_numpy(A).to_sparse()<br>
It would work if your PyTorch version &gt;= 1.0.0</p>

username: zcajiayin
cooked: <p>Right, I realize this is a stupid question. Thanks.</p>
Job crashed in pytorch 1.1.0 because of oom problem

username: K_Li
cooked: <p>hi,all:<br>
when I set data loader’s num_work &gt; 0,I got this error<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/6/96d51312bcd8a090da40b012a67c8c99c9d8c23c.png" data-download-href="https://discuss.pytorch.org/uploads/default/96d51312bcd8a090da40b012a67c8c99c9d8c23c" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/6/96d51312bcd8a090da40b012a67c8c99c9d8c23c_2_690x177.png" alt="image" data-base62-sha1="lwk85YP77LnSOyTETtRAJyBfEYY" width="690" height="177" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/6/96d51312bcd8a090da40b012a67c8c99c9d8c23c_2_690x177.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/6/96d51312bcd8a090da40b012a67c8c99c9d8c23c_2_1035x265.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/6/96d51312bcd8a090da40b012a67c8c99c9d8c23c_2_1380x354.png 2x" data-dominant-color="F4F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1886×484 98 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
it look like out of cpu memory,but when I check memory, it only use 5% memory(but lots of virtual memory)<br>
and I use free command to checkout the memory, i found free part keep decreasing，but buffer/cache part keep increase ,and finally  job will crash</p>
<p>I don’t find this problem when I use pytorch 1.0.0，how can I fix it？and by the way，why it use so many virtual memory？</p>
How to convert mnist images to variables images and labels

username: ati
cooked: <p>Hi<br>
I have a code as below:</p>
<pre><code class="lang-auto">dataset = MNIST(path=data_path, download=True, shuffle=True)

if train:
    images, labels = dataset.get_train()
else:
    images, labels = dataset.get_test()

images, labels = images[:n_examples], labels[:n_examples]
images, labels = iter(images.view(-1, 784) / 255), iter(labels)
</code></pre>
<p>but when i run it, it gives me this error</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "C:\Users\Ati\Downloads\Compressed\bindsnet_experiments-master\experiments\mnist\two_layer_backprop.py", line 135, in &lt;module&gt;
    images, labels = dataset.get_train()
AttributeError: 'TorchvisionDatasetWrapper' object has no attribute 'get_train'
</code></pre>
<p>I think because <code>get_train()</code> is out of date , it doesn’t support by <code>torchvision</code><br>
But i tested different ways for converting mnist data to images and labels variables<br>
Who knows how could i change it when <code>get_train() </code>doesn’t work<br>
I will appreciate your help if someone helps me on this</p>

username: ptrblck
cooked: <p>You could access the underlying data and targets using:</p>
<pre><code class="lang-python">dataset.data
dataset.targets
</code></pre>
<p>Where did you see the <code>get_train</code> method?</p>

username: ati
cooked: <p>Hey…thank you for your response and yeah it did work:smiley:<img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"><img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"><br>
I got that from this link<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/BINDS-LAB-UMASS/bindsnet_experiments/blob/master/experiments/mnist/backprop.py" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/BINDS-LAB-UMASS/bindsnet_experiments/blob/master/experiments/mnist/backprop.py" target="_blank" rel="nofollow noopener">BINDS-LAB-UMASS/bindsnet_experiments/blob/master/experiments/mnist/backprop.py</a></h4>
<pre><code class="lang-py">import os
import torch
import argparse
import numpy as np
import matplotlib.pyplot as plt

from time import time as t
from sklearn.metrics import confusion_matrix

from bindsnet.datasets import MNIST
from bindsnet.network import Network, load_network
from bindsnet.network.monitors import Monitor
from bindsnet.network.topology import Connection
from bindsnet.network.nodes import RealInput, IFNodes
from bindsnet.analysis.plotting import plot_spikes, plot_weights

from experiments import ROOT_DIR

data = 'mnist'
model = 'backprop'
</code></pre>

  This file has been truncated. <a href="https://github.com/BINDS-LAB-UMASS/bindsnet_experiments/blob/master/experiments/mnist/backprop.py" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
Visualizing accuracy & training loss during training

username: Er_Hall
cooked: <p>Hi all,</p>
<p>Can someone tell me which package/module/tool is the most popular/accurate this period in order to visualize accuracy and training loss with PyTorch ?</p>
<p>I have found several questions in this forum about this problem but some discussions are outdated (2 years ago)</p>
<p>Thanks in advance</p>

username: Eta_C
cooked: <p><a href="https://pytorch.org/tutorials/intermediate/tensorboard_tutorial.html" rel="nofollow noopener">VISUALIZING MODELS, DATA, AND TRAINING WITH TENSORBOARD</a></p>
Using DataLoader to load two tensors

username: blade
cooked: <p>I am trying to divide a graph dataset into multiple batches using DataLoader. My input data is consist of a graph signal of size <code>[B, feature * node]</code> and graph adjacency matrix of size <code>[B, node, node]</code>. I was wondering if there is a way to divide both tensor into batches at once using DataLoader.</p>

username: blade
cooked: <p>Found the solution here:</p>
<aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="26605">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"><a href="https://discuss.pytorch.org/t/make-a-tensordataset-and-dataloader-with-multiple-inputs-parameters/26605/2">Make a TensorDataset and Dataloader with multiple inputs parameters</a></div>
<blockquote>
<p>Here is a small example using just random data:</p>
<pre><code class="lang-auto">nb_samples = 100
features = torch.randn(nb_samples, 10)
labels = torch.empty(nb_samples, dtype=torch.long).random_(10)
adjacency = torch.randn(nb_samples, 5)
laplacian = torch.randn(nb_samples, 7)

dataset = TensorDataset(features, labels, adjacency, laplacian)
loader = DataLoader(
    dataset,
    batch_size=2
)

for batch_idx, (x, y, a, l) in enumerate(loader):
    print(x.shape, y.shape, a.shape, l.shape)
</code></pre>
</blockquote>
</aside>
About processing pytorch image segmenation in c++

username: swp
cooked: <pre><code>string model_path = "Deep0.93.pt";
std::ifstream is(model_path, std::ifstream::binary);
std::shared_ptr&lt;torch::jit::script::Module&gt; module = torch::jit::load(is);


module-&gt;to(at::kCUDA);

assert(module != nullptr);
std::cout &lt;&lt; "ok\n";


img_color = imread("psw.jpg", IMREAD_COLOR);
imshow("abc", img_color);

cvtColor(img_color, img_color, COLOR_BGR2RGB);

if (img_color.empty()) {
	//continue;
	return -1;
}

std::vector&lt;int64_t&gt; sizes = { 1, 3, img_color.rows, img_color.cols };
at::TensorOptions options(at::ScalarType::Byte);
at::Tensor tensor_image = torch::from_blob(img_color.data, at::IntList(sizes), options);
tensor_image = tensor_image.toType(at::kFloat);
std::vector&lt;torch::jit::IValue&gt; inputs;
inputs.emplace_back(tensor_image);
at::Tensor results = module-&gt;forward(inputs).toTensor().to(at::kCUDA);

auto output = results.cpu().detach();

output = output.squeeze();


Mat result(output.size(0), output.size(1), CV_32F, output.data_ptr());

Mat temp = result.clone();

for (int i = 0; i &lt; temp.rows; i++)
{
	for (int j = 0; j &lt; temp.cols; j++)
	{
		if (temp.at&lt;float&gt;(i, j) &gt; 0.5f)
			temp.at&lt;float&gt;(i, j) = 255;
		else
			temp.at&lt;float&gt;(i, j) = 0;
	}
}


temp.convertTo(temp, CV_8U);
cv::resize(temp, temp, Size(224, 224), 1);
imshow("result", temp);
waitKey(0);
</code></pre>
<p>This is my code…</p>
<p>but the result is too weird.</p>
<p>Do I have any mistakes?</p>
<p>The model is not a problem</p>
<p>My guess is that the mat type changes strangely in the tensor conversion process.</p>
<p>refer… save pt file</p>
<p>net = net()<br>
net.to(device)</p>
<p>net.load_state_dict(torch.load(…))<br>
…<br>
example = torch.read(1,3,224,224)<br>
example = example.to(device)<br>
traced_script_module = torch.jit.trace(net, example)<br>
output = traced_script_module(example)<br>
traced_script_module.save(“xxx.pt”)</p>

username: Kallinski
cooked: <p>I have a similar problem with a fully convolutional network for image segmentation. Python results are as expected but libtorch c++ frontend predictions differ a lot. Did you find a solution for this problem?</p>

username: yf225
cooked: <p><a class="mention" href="/u/swp">@swp</a> <a class="mention" href="/u/kallinski">@Kallinski</a> Would you like to post the issue with a minimal reproducible example in <a href="https://github.com/pytorch/pytorch/issues" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues</a>?</p>

username: Erun
cooked: <p>Hi, have you solved the problem? I also use the libtorch to do my segmentation task, but the results which are visualized  are strange,  but i  print  the  logits or array of the results , i find the logits is the same as that in pytorch , so  my guess is also the mat type changes strangely in the tensor conversion process.<br>
I hope i can get your reply, thank you so much, i need help urgently…</p>
AttributeError: 'VGG' object has no attribute 'copy'

username: eshan87
cooked: <p>please find code snippet below:-</p>
<pre><code>def load_torch_model(self):
    if self.pre_trained_model == 'VGG16':
        self.torch_model = models.vgg16(pretrained=True)
        print(self.torch_model)
        self.torch_model.classifier[6] = nn.Sequential(nn.Linear(in_features=self.in_features[0],
                                                                 out_features=self.out_features[0], bias=True),
                                                       nn.ReLU(), nn.Dropout(p=self.dropout_ratio[0]),
                                                       nn.Linear(in_features=self.in_features[1],
                                                                 out_features=self.out_features[1], bias=True),
                                                       nn.LogSoftmax(dim=1))
        print(self.torch_model)

    self.torch_model.load_state_dict(torch.load(&lt;Path of model with .pt extension&gt;))
</code></pre>
<p>I got traceback:-</p>
<p>Traceback (most recent call last):<br>
File “C:/AI_Projects/Pytorch_to_TensorFlow_API/test.py”, line 20, in <br>
convert.load_torch_model()<br>
File “C:\AI_Projects\Pytorch_to_TensorFlow_API\converter.py”, line 43, in load_torch_model<br>
self.torch_model.load_state_dict(torch.load(self.torch_model_path))<br>
File “C:\Python37\lib\site-packages\torch\nn\modules\module.py”, line 818, in load_state_dict<br>
state_dict = state_dict.copy()<br>
File “C:\Python37\lib\site-packages\torch\nn\modules\module.py”, line 591, in <strong>getattr</strong><br>
type(self).<strong>name</strong>, name))<br>
AttributeError: ‘VGG’ object has no attribute ‘copy’</p>

username: ptrblck
cooked: <p>How did you save the <code>state_dict</code> and what keys are inside it?<br>
Saving and loading the <code>state_dict</code> using your model, works fine:</p>
<pre><code class="lang-python">torch_model = models.vgg16(pretrained=False)

torch_model.classifier[6] = nn.Sequential(
    nn.Linear(in_features=4096,
        out_features=1, bias=True),
    nn.ReLU(),
    nn.Dropout(p=0.5),
    nn.Linear(in_features=1, out_features=2, bias=True),
    nn.LogSoftmax(dim=1)
)

torch_model(torch.randn(1, 3, 224, 224))
state_dict = torch_model.state_dict()
torch.save(state_dict, 'tmp.pt')

torch_model.load_state_dict(torch.load('tmp.pt'))
</code></pre>