akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Symmetric Padding

fbcotter: <p>Has anybody figured out how to do symmetric padding of an image tensor?</p>
<p>The current pad function can do constant padding, zero-order hold padding (replicate) or reflection padding, but it can’t do symmetric padding. The difference between symmetric and reflection padding is best seen in the tensorflow help:</p>
<pre><code class="lang-python">t = tf.constant([[1, 2, 3], [4, 5, 6]])
paddings = tf.constant([[1, 1,], [2, 2]])
# 'constant_values' is 0.
# rank of 't' is 2.
tf.pad(t, paddings, "CONSTANT")  # [[0, 0, 0, 0, 0, 0, 0],
                                 #  [0, 0, 1, 2, 3, 0, 0],
                                 #  [0, 0, 4, 5, 6, 0, 0],
                                 #  [0, 0, 0, 0, 0, 0, 0]]

tf.pad(t, paddings, "REFLECT")  # [[6, 5, 4, 5, 6, 5, 4],
                                #  [3, 2, 1, 2, 3, 2, 1],
                                #  [6, 5, 4, 5, 6, 5, 4],
                                #  [3, 2, 1, 2, 3, 2, 1]]

tf.pad(t, paddings, "SYMMETRIC")  # [[2, 1, 1, 2, 3, 3, 2],
                                  #  [2, 1, 1, 2, 3, 3, 2],
                                  #  [5, 4, 4, 5, 6, 6, 5],
                                  #  [5, 4, 4, 5, 6, 6, 5]]
</code></pre>

Felipe_bivort_haiek: <p>Do you know if there haas been any way to use convnets with symmetric padding in pytorch?</p>

fbcotter: <p>Hi <a class="mention" href="/u/felipe_bivort_haiek">@Felipe_bivort_haiek</a>, I managed to find a workaround before. You can achieve symmetric padding by manually making the index arrays. For example, this function will do 2d padding for you:</p>
<pre><code class="lang-auto">import torch
import numpy as np
from typing import Tuple

def symm_pad(im: torch.Tensor, padding: Tuple[int, int, int, int]):
     h, w = im.shape[-2:]
     left, right, top, bottom = padding
 
     x_idx = np.arange(-left, w+right)
     y_idx = np.arange(-top, h+bottom)
 
     def reflect(x, minx, maxx):
         """ Reflects an array around two points making a triangular waveform that ramps up
         and down,  allowing for pad lengths greater than the input length """
         rng = maxx - minx
         double_rng = 2*rng
         mod = np.fmod(x - minx, double_rng)
         normed_mod = np.where(mod &lt; 0, mod+double_rng, mod)
         out = np.where(normed_mod &gt;= rng, double_rng - normed_mod, normed_mod) + minx
         return np.array(out, dtype=x.dtype)

     x_pad = reflect(x_idx, -0.5, w-0.5)
     y_pad = reflect(y_idx, -0.5, h-0.5)
     xx, yy = np.meshgrid(x_pad, y_pad)
     return im[..., yy, xx]
</code></pre>
Activation function in output layer of autoencoders

Aritra: <p>Do we need to use an activation function on the final decoding layer of an autoencoder?</p>

acobobby: <p>It depends on the loss function you are using. If the loss takes logits in input, then it most likely implements the appropriate nonlinearity and you can use just a linear layer as your decoder output.<br>
If you use a custom loss, you may have to use an activation function.</p>

Aritra: <p>What if I’m using an autoencoder with MSE loss?<br>
Should I use linear activation for the final decoding layer?</p>

acobobby: <p>Since you are using MSE, you are probably dealing with regression task (predicting real valued output features). Then, you can use just a linear layer as output layer.<br>
The model will learn to produce values according to the targets you provide to it.</p>
<p>Just to be clear: if you were dealing with a classification task, in principle you should have used softmax activation function in order to restrict your output in a probability space and then pick the most probable one as predicted class. But since the softmax function is already implemented in the CrossEntropyLoss (used for classification tasks), you watned to use only a linear layer also in this case.</p>

Aritra: <p>Thanks for explaining</p>
Use adjacency matrix to sum tensors' values with scatter_add

giannipele: <p>Hello. Is there a simple way to sum 2D tensors’ values with torch.scatter_add() by starting from the adjacency matrix? This is an example explaining what I would like to obtain:</p>
<pre><code class="lang-auto">x = torch.tensor([[1., 2.],
                  [3., 4.],
                  [5., 6.],
                  [7., 8.]])

adj = torch.tensor([[1, 1, 0, 1],
                    [1, 1, 1, 0],
                    [0, 1, 1, 0],
                    [1, 0, 0, 1]])
</code></pre>
<p>The output tensor after the operations should be</p>
<pre><code class="lang-auto">[[11., 14.],
 [9., 12.],
 [8., 10.], 
 [8., 10.]]
</code></pre>
<p>Meaning that sums should be computed in this way:</p>
<pre><code class="lang-auto">[ [1+3+7, 2+4+8],
  [3+1+5, 6+2+4],
  [5+3, 6+4],
  [7+1, 8+2]]
</code></pre>
<p>I remark that solving this with scatter_add() is quite important (but, of course, I am open to other solutions!).</p>

tom: <p>It’s not scatter_add, but your task is essentially matrix multiplication <code>adj.float() @ x</code>.</p>
<p>Best regards</p>
<p>Thomas</p>

giannipele: <p>Ops, right, I didn’t see it at the beginning. Thanks!</p>
Is attention useful for classification tasks?

Peter_Featherstone: <p>Are attention modules like SCSE useful for classification?<br>
I’ve seen them being used in decoder blocks of segmentation models but not in classification models? I have a classification problem (not in the image domain) and no matter what model i use, i can’t get passed ~65% accuracy. I was wondering if self-attention would be useful but noticed that it’s nowhere used in classification tasks. Any idea why?</p>
Mixed precision training with Apex

a_d: <p>Hello all,<br>
I am trying to convert my model to half-precision followed by mixed-precision training so that lesser memory could be used. I am not sure whether this is the correct way to do it -</p>
<pre><code class="lang-auto">
    def train_model(self, model, dataloader, epochs):
        cudnn.benchmark = True
        optimizer = torch.optim.SGD(model.parameters(),lr=0.01)
        model.half()
        model ,optimizer = amp.initialize(model, optimizer, opt_level='01')
        model.train()
        model.cuda()
        scheduler = torch.optim.lr_scheduler.StepLR(optimizer=optimizer, step_size=10, gamma=0.1)
        #criterion = torch.nn.CrossEntropyLoss().cuda()
        criterion = torch.nn.BCELoss().cuda()
        for i in range(0, epochs):
            scheduler.step()
            train_accuracy = 0
            net_loss = 0
            for _, (data, label) in enumerate(dataloader):
                optimizer.zero_grad()
                data = data.half().cuda()
                label = label.cuda()
                out = model(data)
                loss = criterion(out, label)
                with amp.scale_loss(loss,optimizer) as scaled_loss:
                    scaled_loss.backward()
                    optimizer.step()
                if torch.argmax(out) == label:
                    train_accuracy +=1
                net_loss += loss.item()
            print('------------------------------------------')
            print('EPOCH ', i)
            print(train_accuracy/len(dataloader))
            print(net_loss/len(dataloader))
</code></pre>
<p>Could someone please guide me in the correct direction<br>
Thanks in Advance</p>

ptrblck: <p><code>nn.BCELoss</code> is most likely unsafe to be used in mixed-precision training and you should get a warning.<br>
Besides that, you shouldn’t call <code>model.half()</code> and push the model to the device before initializing <code>amp</code>.<br>
The <a href="https://nvidia.github.io/apex/amp.html">docs</a> should give you an example usage.</p>
<p>That being said, we recommend to try out <a href="https://pytorch.org/docs/stable/amp.html">torch.cuda.amp</a> using the nightly binaries or by installing from source, as it can be used directly in PyTorch now without the need to build <code>apex</code>.</p>
Error when training

Imene.b: <p>whene i try to train my neural network i meet this error:<br>
torch/nn/modules/module.py", line 477, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/imene/anaconda3/envs/pose/lib/python3.6/site-packages/torch/nn/modules/conv.py”, line 301, in forward<br>
self.padding, self.dilation, self.groups)<br>
RuntimeError: CUDA error: out of memory<br>
I tried to put batch size to 1 but that does not work,<br>
I have an i7 RAM = 12G, GPU = [GeForce GTX 750 Ti] (rev a2) 2G<br>
can any one give me solution because i am blocked since tow weeks<br>
tanks.</p>

ptrblck: <p>2GB might not be enough for your training to run.<br>
You could try to trade compute for memory using <code>torch.utils.checkpoint</code> or just push some layers to the GPU and keep others on the CPU.</p>

Imene.b: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="78527">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>to trade compute for memory using <code>torch.utils.checkpoint</code></p>
</blockquote>
</aside>
<p>tanks for your replay, have you any example to how to do this because i am novice in using pytorch</p>

ptrblck: <p>Have a look at <a href="https://github.com/prigoyal/pytorch_memonger/blob/master/tutorial/Checkpointing_for_PyTorch_models.ipynb">this notebook</a>. Note that it’s quite old by now and you wouldn’t need to use <code>Variables</code> anymore.<br>
However, it might still give you a good idea about the usage of checkpointing.</p>

Imene.b: <p>tanks i will read it</p>
Pytorch indexing

Luca_Pamparana: <p>So given a  torch tensor of a given shape say (<code>[1, 1, 10, 11, 12]</code>) and another torch tensor created as:</p>
<pre><code class="lang-auto">rng = torch.tensor([9, 10, 11])
</code></pre>
<p>Now, I want to use this range over the last 3 dimensions. So something which statically would be equivalant to:</p>
<pre><code class="lang-auto">x = torch.rand(1, 1, 10, 11, 12)
rng = torch.tensor([9, 10, 11])

# statically want something like:

x[:, :, :rng[0], :rng[1], :rng[2]]
</code></pre>
<p>However, the dimension of the input tensor is dynamic  and I wonder if there is a way to do this kind  of indexing dynamically using torch? Otherwise, I have to basically hard code the input dimensions which is ugly and difficult to maintain.</p>

ptrblck: <p>You could use <code>x[..., :rng[0], :rng[1], :rng[2]]</code> to index the last dimensions.<br>
Let me know, if that would work for you.</p>

Luca_Pamparana: <p>The problem is <code>rng</code> can be dynamic, so it can be 1D, 2D, 3D or 4D. If I do this, I have to hard code the dimensions. i.e. something like:</p>
<pre><code class="lang-auto">if len(rng) == 1:
    ....
elif len(rng) ==  2:
   ...
</code></pre>
<p>Is there a way to do this indexing directly with the tensor object?</p>

ptrblck: <p>I misunderstood this part:</p>
<aside class="quote no-group" data-username="Luca_Pamparana" data-post="1" data-topic="78559">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/luca_pamparana/48/14310_2.png" class="avatar"> Luca_Pamparana:</div>
<blockquote>
<p>However, the dimension of the input tensor is dynamic</p>
</blockquote>
</aside>
<p>as <code>x</code> would be the input tensor.</p>
<p>Note that <code>rng</code> is a 1-dimensional tensor with 3 values.<br>
If you want to index the last 3 values, you could use negative indices:</p>
<pre data-code-wrap="python"><code class="lang-python">x[:, :, :rng[-3], :rng[-2], :rng[-1]]
</code></pre>
Custom submodules in libtorch C++

jackeown: <p>Can you show an example of using <code>register_module</code> with a custom module?<br>
The only examples I’ve found online are using linear layers or convolutional layers.</p>
<p>I tried to write my own module and register it with another module and I couldn’t get it to work.<br>
My IDE is telling me <code>no instance of overloaded function "MyModel::register_module" matches the argument list -- argument types are: (const char [14], TreeEmbedding)</code></p>
<p><code>TreeEmbedding</code> is the name of another struct I made which extends <code>torch::nn::Module</code>.</p>
<p>Am I missing something? An example of this would be very helpful.</p>
<p>Thanks,<br>
Jack</p>
Using Batchsampler to fetch batched data from a remote dataset

dvirginz: <p>I have a need to use a <code>BatchSampler</code> that is exactly as described in the Pytorch documentation, yet, I cannot understand how to use the batchsampler with <strong>any</strong> given dataset.</p>
<p>e.g</p>
<pre><code class="lang-auto">class MyDataset(Dataset):

    def __init__(self, remote_ddf, ):
        self.ddf = remote_ddf

    def __len__(self):
        return len(self.ddf)

    def __getitem__(self, idx):
        return self.ddf[idx] --------&gt; This is as expensive as a batch call

    def get_batch(self, batch_idx):
        return self.ddf[batch_idx]

my_loader = DataLoader(MyDataset(remote_ddf), 
           batch_sampler=BatchSampler(Sampler(), batch_size=3))

</code></pre>
<p>The thing I do not understand, neither found any example online or in torch docs, is how do I use my get_batch function instead of the __getitem__ function.<br>
Practically, I would even prefer to not implement the __getitem__, and let my custom dataset create the whole batched sample.</p>
Gumbel softmax change his value in an unresonable (for me) way

barakb: <p>Hi, I’ll show what I meant by an example, I’ll first define:</p>
<pre><code class="lang-auto">import torch
from torch.nn import functional as F
import numpy as np
import random
#####
torch.manual_seed(60)
torch.cuda.manual_seed(60)
np.random.seed(60)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
random.seed(60)
#####
</code></pre>
<p>Now let’s do something simple:</p>
<pre><code class="lang-auto">pred = torch.FloatTensor([0,1,2])
a = F.gumbel_softmax(pred)
b = F.gumbel_softmax(pred)
print("a:",a)
print("b:",b)
</code></pre>
<p>we will get:</p>
<blockquote>
<p>a: tensor([0.4957, 0.1406, 0.3637])<br>
b: tensor([0.0653, 0.6350, 0.2996])</p>
</blockquote>
<p>Now if I remove a:</p>
<pre><code class="lang-auto">pred = torch.FloatTensor([0,1,2])
b = F.gumbel_softmax(pred)
print("b:",b)
</code></pre>
<p>we will get:</p>
<blockquote>
<p>b: tensor([0.4957, 0.1406, 0.3637])</p>
</blockquote>
<p>And it continues as follows with 3 instances.</p>
<p>I understand that it is depended on the seed, but it is impossible that doing humble softmax on a vector is deepened on if I did it before and stored it on another variable.</p>
<p>Moreover that doing twice gumbel softmax as follows:</p>
<pre><code class="lang-auto">pred = torch.FloatTensor([0,1,2])
a = F.gumbel_softmax(pred)
b = F.gumbel_softmax(pred)
print("a:",a)
print("b:",b)
</code></pre>
<p>Consistantly showed better results on multiple seeds.</p>
<p>Can anyone help me understand this behaviour!?</p>

barakb: <p>Pinging this is super important.</p>
An error occured when use pretrained resnet50 encode images

mhxin: <p>i just want to get the encode of images use pretrained resnet50, Here is my code</p>
<pre><code class="lang-auto">from torchvision.models.resnet import resnet50
import torch.nn as nn
import torch as t
from utils import preprocess

def resnet():
    net = resnet50(pretrained=True)
    # delete fc layer
    del net.fc
    for param in net.parameters():
        param.requires_grad = False

    # print(net)
    return net

model = resnet()

img = preprocess(img_name)

# img(tensor) -&gt; [1, 3, 299, 299]
encode_img = model(img)
</code></pre>
<p>when i run above code, it give me an error like this.</p>
<p>Exception has occurred: AttributeError<br>
‘ResNet’ object has no attribute ‘fc’</p>
<p>how could i solve this problem, if anyone can tell me. I will be very grateful</p>

ptrblck: <p>If you delete <code>net.fc</code>, the <code>forward</code> pass will still try to call it and will thus raise this error.<br>
To “disable” this layer, just replace it with <code>nn.Identity()</code>.</p>

mhxin: <p>thank you very much, this is a new kownledge for me <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
Optimizing combination of two loss function with a negative value and a positive value

amirhf: <p>I am trying to get my head around if optimizing a combination two loss function with one with positive values and the other with negative values is possible. Basically, the problem is to minimize f = f1 + f2 where f1 has positive values and f2 has negative values. So minimizing f is equivalent to minimizing f1 (getting close to zero) and maximizing -f2 (which means minimizing f2 and getting larger negative values). Specifically, the flow of data in my classification problem is as below:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/0/80cd59c2c5ec98703371ff94e91a1675c376440e.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/80cd59c2c5ec98703371ff94e91a1675c376440e" title="File_001"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/0/80cd59c2c5ec98703371ff94e91a1675c376440e_2_690x287.jpeg" alt="File_001" data-base62-sha1="inr4PtZSFa9Xqao0FooNYIptcdg" width="690" height="287" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/0/80cd59c2c5ec98703371ff94e91a1675c376440e_2_690x287.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/8/0/80cd59c2c5ec98703371ff94e91a1675c376440e_2_1035x430.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/0/80cd59c2c5ec98703371ff94e91a1675c376440e.jpeg 2x" data-dominant-color="F3F2ED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">File_001</span><span class="informations">1320×550 167 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
where xi is the feature for one input (network output). CE is crossentropy loss. My custom loss is a custom function of xi and some additional parameters that outputs negative values and total loss = CE + alpha * CustomLoss.</p>
<p>I’m eager to know your comments on this model.</p>
How to judge scalar is equal to zero or one, etc

tgsmdww: <p>I want to know a scalar is equal to one or not. How to write such code ?</p>

russellizadi: <pre><code class="lang-auto">scalar = torch.tensor(1)
assert scalar==1
# or
if scalar==1:
    print('yes')
</code></pre>

tgsmdww: <p>It will report an error</p>
<p>—Original—</p>
MultiLabelMarginLoss definition

ariaaay: <p><code>loss(x, y) = sum_ij(max(0, 1 - (x[y[j]] - x[i]))) / x.size(0)</code><br>
Here is the definieition given by Pytorch for its <a href="http://pytorch.org/docs/master/nn.html#torch.nn.functional.multilabel_soft_margin_loss" rel="nofollow noopener">multilabel margin loss</a>, "where i == 0 to x.size(0), j == 0 to y.size(0), y[j] != 0, and i != y[j] for all i and j."<br>
It doesn’t make sense to me at all since hinge loss should be something like L(y) = Sum_ij max(0, 1-x[j]*y[j]). Could anyone clarify this?<br>
Also I would really appreciate if anyone could find the source code of this loss function.<br>
Thanks!</p>

SpandanMadan: <p>Are you trying to do Multi Label Classification? If so, use BCELoss, not Multi label margin loss. I will be uploading a tutorial on the same soon, as a lot of people are having problems with multi label.</p>

lidan: <p>Hi! I wonder if you uploaded the tutorial on multi-label classification? I could not find it. Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

SpandanMadan: <p>Hi, I thought some one else did so I didn’t do it. Can you try looking in this forum and see if you find it? If not I can write one whenever I get some time (probably next month).</p>

lidan: <p>Thanks.<br>
I think there is no tutorial on this. But I do find some discussions on multi-label classification.<br>
Could you please give an explanation that why should we use BCELoss rather than Multi label margin loss?</p>

sudddddd: <p>Isn’t BCELoss for binary classification only?</p>

russellizadi: <p>It could be used in non-binary classification when targets are so-called one-hot vectors. The term binary comes from its two-term definition.</p>
Different behivors between pytorch1.0.1 and 1.3.1

Mabinogiysk: <p>I am working on training siamfc on pytorch 1.3.1.<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="16" height="16">
      <a href="https://github.com/huanglianghua/siamfc-pytorch" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars0.githubusercontent.com/u/8928997?s=400&amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/huanglianghua/siamfc-pytorch" target="_blank" rel="nofollow noopener">huanglianghua/siamfc-pytorch</a></h3>

<p>A clean PyTorch implementation of SiamFC tracking/training, evaluated on 7 datasets. - huanglianghua/siamfc-pytorch</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>The results in paper were reproducing using pytorch 1.0.1 with AUC 0.57, however, the AUC in pytorch 1.3.1 is about 0.52.<br>
I also trained model in pytorch 1.0.1 and load it in pytorch 1.3.1, the results is still bad. I found there is slightly difference in the outputs of backbone on some images.<br>
I have no idea what cause this difference because it is a very simple network，is there any suggestions?</p>
Get the input channels for the conv2d from previous layer?

Aayush_Garg: <p>Hi guys,<br>
I was wondering if there are many convolutional layers (conv1 --&gt; conv2 ). How can we get the input channels parameter for the conv2 from the conv1 output channel?</p>
<pre><code class="lang-auto">class MyModel(nn.Module):
  def __init__(self, in_ch, num_features, out_ch2):
    super(MyModel, self).__init__()
    self.conv1 = nn.Conv2D(in_channels,num_features)
    self.conv2 = nn.Conv2D(in_channnels_from_out_channels_of_conv1,out_ch2)
</code></pre>
<p>Can I get the out_channels from the conv1 layer and use it as in_ch for conv2?</p>

ptrblck: <p>Yes, you can directly access this property via:</p>
<pre><code class="lang-python">self.conv1.out_channels
</code></pre>
<p>For your code snippet, this should work:</p>
<pre><code class="lang-python">self.conv1 = nn.Conv2D(in_channels,num_features)
self.conv2 = nn.Conv2D(self.conv1.out_channels,out_ch2)
</code></pre>

Aayush_Garg: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thanks,  I dont think I framed my question correctly earlier.</p>
<p>Actually what I am looking for  is something this</p>
<pre><code class="lang-auto"># Class for a single Block
class SingleBlock(nn.Module):

  def __init__(self, in_channels, num_channels, drop_out=0.2):
    super(SingleBlock, self).__init__()

    self.conv = conv3X3(in_channels, num_channels)
#    self.outchannels = self.conv.out_channels

# def get_out_channels(self):
#   return _outchannels
  
  def forward(self, x):
    concat = x
    out = self.conv(out)
    out = torch.cat((out,concat), axis=1)

    ## Only while Q.C. to see output shape
    # out = self.lrelu(out)

    return out

# Q. C. single SingleBlock
qc_toggle = 1
if qc_toggle == 1:
  
  # print the model summary (Not exactly great)
  temp_block = DenseBlock(3,32)
  print(temp_block)
</code></pre>
<p>When I make this network I want to access the out_channels in this (after the torch.cat) so that I can use it in some other place.</p>
<p>Can I do some get function or is there any property to get the out_channels in this network?</p>
Converting tensorflow cast function with condition in pytorch | Kernel Crashes

_joker: <p>I’m looking for an equivalent of TensorFlow in Pytorch</p>
<pre><code class="lang-auto">binary_test = tf.cast(binary &gt; 0.5, tf.float32)
</code></pre>
<p>I tried;</p>
<pre><code class="lang-auto">binary_test = binary.type(float32)
binary_test = torch.where(binary_test &gt; 0.5, x, y)
</code></pre>
<p>where,   <code>x = Tensor([1]), y = Tensor([0])</code></p>
<p>On doing so the python kernel crashes.</p>

ptrblck: <p>The posted code should generally work:</p>
<pre><code class="lang-python">binary = torch.randint(0, 2, (10,)).byte()
binary_test = binary.type(torch.float32)
binary_test = torch.where(binary_test &gt; 0.5, torch.tensor([1.]), torch.tensor([0.]))
</code></pre>
<p>Note that I had to fix the <code>float32</code> to <code>torch.float32</code> and am not sure if it’s a copy-paste issue or if you might be running into this error.</p>
<p>Could you post the values of <code>binary</code>, which create this error please?</p>
Dropout during inference

Luca_Pamparana: <p>I would like to enable dropout during inference. So, I am creating the dropout layer as follows:</p>
<pre><code class="lang-auto">self.monte_carlo_layer = None
if monte_carlo_dropout:
    dropout_class = getattr(nn, 'Dropout{}d'.format(dimensions))
    self.monte_carlo_layer = dropout_class(p=monte_carlo_dropout)
</code></pre>
<p>And I invoke it in the <code>forward</code> function as:</p>
<pre><code class="lang-auto">def forward(self, x):
   ...
   if self.monte_carlo_layer is not None:
       x = self.monte_carlo_layer(x)
</code></pre>
<p>My question is will this ensure that the dropout will be invoked even during testing i.e. with <code>eval</code> activated?</p>

ptrblck: <p>I’m not sure how the posted code is used, but would recommend to explicitly set <code>train()</code> on the dropout module via:</p>
<pre><code class="lang-python">model.eval() # sets all layers to eval
model.drop_layer.train() # resets dropout to train
</code></pre>
How to make DataLoader use batched indexing?

StrausMG: <p>Hi! I read <a href="https://pytorch.org/docs/stable/data.html#disable-automatic-batching" rel="nofollow noopener">this</a> section several times and tried to google my question, still cannot figure it out.</p>
<p>My dataset is a tensor that fits in memory, so using batched indexing is much more efficient than indexing items one by one and collating them. Below, I provide two solutions I came up with, but is there a better way?</p>
<h3>Option A</h3>
<p>In this case I don’t fully utilize <code>worker_count</code> and <code>pin_memory</code>. I didn’t measure if this is important in my case, but anyway:</p>
<pre><code class="lang-python">class IndicesDataset(Dataset):
    def __init__(self, size):
        self.size = size

    def __len__(self):
        return self.size

    def __getitem__(self, index):
        return index


for batch_idx in DataLoader(IndicesDataset(size), *args, **kwargs):
    x, y = X[batch_idx], Y[batch_idx]
</code></pre>
<h3>Option B</h3>
<p>Make a dataset with <code>__getitem__</code> that is aware of batches. Then:</p>
<pre><code class="lang-python">dataloader = DataLoader(dataset, batch_size=None, sampler=BatchSampler(...))
</code></pre>
<p>where BatchSampler yields <strong>lists of indices</strong> (which directly contradicts to the meaning of the <code>sampler</code> argument, I know).</p>

ptrblck: <p>Both approaches would most likely work, although I think you could remove the <code>DataLoader</code> from the first approach and add shuffling manually, if wanted.</p>
<p>If you have already loaded the data into the RAM, you could directly get the samples via indexing or <code>gather</code>.</p>
Mapping function to tensor

sapo: <p>Hello, I was just wondering why in PyTorch there is no tool to map functions to<br>
tensor elements, something like<br>
<a href="https://www.tensorflow.org/api_docs/python/tf/map_fn?version=stable" rel="nofollow noopener">tf.map_fn</a><br>
or<br>
<a href="https://www.tensorflow.org/api_docs/python/tf/while_loop?version=stable" rel="nofollow noopener">tf.while_loop</a>.</p>
<p>Looping big tensors with native python loops is very slow because each<br>
iteration needs CPU computations. Having a tool like this, wouldn’t make everything easier?</p>

albanD: <p>Hi,</p>
<p>If the functions you give to these tools are also python function, then they will be slow as well. The iteration over the loop won’t be much slower than the body of the loop right?<br>
The reason they exist I think is that tensorflow does not allow you to use python for loops (in tf1 at least) and so you need these special constructs to be able to do loops. In pytorch, you can simply use regular python code, you don’t need these special constructs.</p>

jackeown: <p>I would still like to see these functions in pytorch only because heavy use of them would be faster than native python for loops</p>
Don't know where to start

Daniel_Marshall: <p>So, first off, I’m a touch thinker.  I can’t just read to learn much, I have to actually write code to figure out what’s happening.  My problem is I’ve figured out linear regression easily enough (it’s not terribly complicated using torch), but everything I read moves on immediately to computer vision, which I really don’t care about.  All I really want to do is figure out what’s really next after linear regression (I don’t know what question to ask in google).</p>
<p>Here’s what I want to do - instead of the basic y = 2x +1 problem, I want to add a second variable.  z = 2x * y, for example (I eventually want to build a model for 10 or more variables, but I need somewhere to start).</p>
<p>Can someone tell me what question to ask?  Is this RNN, DNN?  Log?  I just need a pointer or an example that doesn’t involve data loaders or CV.  Just raw numbers.</p>
<p>Appreciate any help I can get, thanks.</p>

TinfoilHat0: <p>Well you seem to be asking for polynomial regression.</p>
<p>In general, there are bunch of nice ML courses at Coursera. They might help you get started.</p>
How to check input size is a multiple of some integer

Luca_Pamparana: <p>I ma currently using something like this to assert that the spatial dimensions of my input tensor is a multiple of 32. My spatial dimensions can be 2D or 3D</p>
<pre><code class="lang-auto">shape = x.shape[2:]  # Do not care about N and C dims.
for s in shape:
    assert s % 32 == 0
</code></pre>
<p>This looks and feels quite ugly and un-pythonic. I was wondering if there is a canonical way to  do this?</p>

Luca_Pamparana: <p>Ok, I figured it out. I needed to convert it to a tensor first.</p>
<pre><code class="lang-auto">shape = torch.tensor(x.shape[2:])
assert torch.sum(shape  % 32) == 0</code></pre>
Having OSError after installing pytorch

otubela_sodiq: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/a/da53147f82459686be0a163bc08cd9531a231a2a.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/da53147f82459686be0a163bc08cd9531a231a2a" title="20200426_114542"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/a/da53147f82459686be0a163bc08cd9531a231a2a_2_375x500.jpeg" alt="20200426_114542" data-base62-sha1="v9o5nC56odwoKSoBElZiGidywDU" width="375" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/a/da53147f82459686be0a163bc08cd9531a231a2a_2_375x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/d/a/da53147f82459686be0a163bc08cd9531a231a2a_2_562x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/a/da53147f82459686be0a163bc08cd9531a231a2a_2_750x1000.jpeg 2x" data-dominant-color="808282"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">20200426_114542</span><span class="informations">1560×2080 736 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>How did you install PyTorch?<br>
Could you also post the stack trace directly by wrapping it into three backticks ``` please? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Libtorch dataset issue

jackeown: <p>I’m trying to build a C++ libtorch model. Following examples I found online, I thought I should use a Dataset to make a data loader to load in batches of my data. However, in the examples I’ve seen online, the “get” method of a custom dataset returns <code>Example&lt;&gt;</code> which assumes that both the input and labels are tensors. What if I want them to be <code>vector&lt;vector&lt;int&gt;&gt;</code>? (A jagged 2d array)</p>
<p>Is this possible? If so, can you give a code example of doing this with libtorch?</p>
<p>Thanks,<br>
Jack</p>

Hartbook: <p>From the doc : <a href="https://pytorch.org/cppdocs/api/classtorch_1_1data_1_1datasets_1_1_stateful_dataset.html#exhale-class-classtorch-1-1data-1-1datasets-1-1-stateful-dataset" class="inline-onebox" rel="noopener nofollow ugc">Template Class StatefulDataset — PyTorch main documentation</a></p>
<blockquote>
<p>Note that when subclassing a from StatefulDataset&lt;Self, T&gt;, the return type of get_batch(), which the subclass must override, will be optional (i.e. the type specified in the StatefulDataset specialization is automatically boxed into an optional for the dataset’s BatchType).</p>
</blockquote>
<p>So you should define your dataset as</p>
<pre><code class="lang-auto">class YourDataset : public torch::data::datasets::StatefulDataset&lt;YourDataset,std::vector&lt;std::vector&lt;int&gt;&gt;&gt;
</code></pre>
<p>And in that case, get_batch will return <code>c10::optional&lt;std::vector&lt;std::vector&lt;int&gt;&gt;&gt;</code></p>
<p>I took StatefulDataset as a base class because I don’t know what kind of dataset you need, but it should work the same for the other types.</p>

jackeown: <p>Thanks for the reply!</p>
<p>In the examples that I found online (<a href="https://github.com/pytorch/examples/blob/master/cpp/custom-dataset/custom-dataset.cpp" rel="nofollow noopener">https://github.com/pytorch/examples/blob/master/cpp/custom-dataset/custom-dataset.cpp</a>), a batch returned from a dataloader had <code>data</code> and <code>target</code> properties. When I use the approach you mention above, the returned batch is of type <code>std::vector&lt;MyClassForExample&gt;</code> This makes sense I guess…since that’s what I told it to return, I’m just curious about how the properties of the <code>Example</code> class (<code>data</code> and <code>target</code>) are available in a returned batch otherwise?</p>
<p>Does the map method of dataset have anything to do with that? What exactly does map do? Specifically what does it do with the Stack transform. Why do people need to “stack” their examples if the make_data_loader takes in the batch size as well…isn’t that the same kind of thing?</p>
<p>Thanks again,<br>
Jack</p>
Fisher information matrix

James_e: <p>Hi,</p>
<p>How would I calculate the Fisher information matrix for a single layer in the network i.e just one nn.Linear</p>
<p>Thanks for your help</p>

russellizadi: <p>Please take a look at the <a href="https://github.com/wiseodd/natural-gradients" rel="nofollow noopener">wiseodd/natural-gradients</a> repository. It has a couple of examples including a 2-layer NN which you can simplify to a single-layer NN.</p>
Libtorch:how to create tensor from tensorRT fp16 half type pointer?

Edwardmark: <p>how to create tensor from tensorRT fp16 half type pointer in libtorch?<br>
I am working on a detection model. I change the backbone of it to tensorRT to do FP16 inference, and the detection code such as decode boxes and nms is done in libtorch and torchvisoin, so how to create fp16 tensor from tensorRT half type pointers?<br>
The important code is to illustrate the issue:</p>
<pre><code class="lang-auto">// tensorRT code to get half type outpus
half_float::half* outputs[18];
doInference(*engine, data, outputs, 1);
// to get the final outputs with libtorch
vector&lt;torch::Tensor&gt; output;
//???? how to feed the date in outpus to output????
// get the result with libtorch method detect_trt-&gt;forward
 auto res = detect_trt-&gt;forward(output); 
</code></pre>
<p>Thanks in advance.</p>

glaringlee: <p>Let’s discuss this here:<br>
<aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/37201" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewbox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/37201" target="_blank" rel="nofollow noopener">Libtorch:how to create tensor from tensorRT fp16 cuda half type pointer?</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2020-04-24" data-time="02:19:45" data-timezone="UTC">02:19AM - 24 Apr 20 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/Edwardmark" target="_blank" rel="nofollow noopener">
          <img alt="Edwardmark" src="https://avatars1.githubusercontent.com/u/15247551?v=4" class="onebox-avatar-inline" width="20" height="20">
          Edwardmark
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">how to create tensor from tensorRT fp16 half type pointer in libtorch?
I am working on a detection model. I change the...</p>
</div>

<div class="labels">
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">module: cpp</span>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
Weight Clipping in a classifier

Angry_potato: <p>HI,<br>
I have implemented a classifier using nn Module and it runs well,but now i want to clip it’s weight between certan value.Kindly tell me the way to do it</p>

TinfoilHat0: <p>Hi,</p>
<p>See the answer from here: <a href="https://discuss.pytorch.org/t/how-to-do-constrained-optimization-in-pytorch/60122" class="inline-onebox">How to do constrained optimization in PyTorch</a></p>
TorchScript and ModuleList Type

Jon2718: <p>Hi,</p>
<p>I am trying to script a PyTorch model.  I am running into the following issues;</p>
<p>My original code utilizes subscripts as follows:</p>
<p>for i in range(self.n_layers):</p>
<pre><code>            self.in_layers[i](a),
            self.cond_layers[I](b) ...
</code></pre>
<p>Both in_layers and cond_layers are of type torch.nn.ModuleList().</p>
<p>When trying to compile I receive an error that these types are not subscriptable.  To work around this issue, I created a new function that receives a torch.nn.ModuleList and iterates over each item to find the right one. I  am trying to use MyPy annotations to get this to compile as follows:</p>
<p>def moduleListChoice(self, ml, choice):<br>
# type: (torch.nn.ModuleList, int)<br>
moduleNumber = 0<br>
for module in ml:<br>
if moduleNumber == choice:<br>
return module<br>
moduleNumber = moduleNumber + 1<br>
print(“Module not found”)</p>
<p>The problem is that I can’t figure out what the type should be for the first argument.  torch.nn.ModuleList doesn’t work.  Any thoughts on how get this to work?</p>
<p>Thanks.</p>

ptrblck: <p><a href="https://github.com/pytorch/pytorch/pull/29236">This PR</a> should have added the support for indexing into <code>nn.ModuleList</code>. Which PyTorch version are you using? If you are using an older one (pre <code>1.5</code>), could you update to the latest stable version and rerun your script, please?</p>

Jon2718: <p>Hi.  Thanks.  I will upgrade to 1.5 and report back.  I’m on 1.4 currently.</p>
ModuleList Object Not Subscriptable

Jon2718: <p>Hi,</p>
<p>I am trying to script some PyTorch code that utilizes ModuleList.  The code makes use of subscripting of a ModuleList object as follows:</p>
<p>or i in range(self.n_layers):<br>
acts = fused_add_tanh_sigmoid_multiply(<br>
self.in_layers<a>i</a>,<br>
self.cond_layers<a>i</a>,<br>
torch.IntTensor([self.n_channels]))</p>
<p>I am met with the following error:</p>
<p>RuntimeError:<br>
‘module’ object is not subscriptable:<br>
at /workspace/tacotron2/waveglow/model.py:147:16<br>
for i in range(self.n_layers):<br>
acts = fused_add_tanh_sigmoid_multiply(<br>
self.in_layers<a>i</a>,<br>
~~~~~~~~~~~~~~~~ &lt;— HERE<br>
self.cond_layers<a>i</a>,<br>
torch.IntTensor([self.n_channels]))</p>
<p>Any help would be greatly appreciated.</p>
<p>Thanks.</p>

ptrblck: <p>Is this a double post from <a href="https://discuss.pytorch.org/t/torchscript-and-modulelist-type/78376/2">here</a> or are these errors differently in some sense?</p>

Jon2718: <p>Thanks.  I will update to 1.5 and try again.</p>
PyTorch and other libraries on Ubuntu 20.04

TGI_Fernando: <p>Has anyone installed PyTorch and other machine learning libraries (sklearn, matplotlib, Pandas, etc.) on Ubuntu 20.04? Please let me know if they are working. Thanks.</p>

Diego_Masotti: <p>Hi, I installed pytorch with Ubuntu 20.04 and everything works except cuda. In 19.10 all worked perfectly, but now pytorch fails to find the gpu.<br>
My machine is an XPS 15 9570 with a 1050ti,with standard installation and drivers.</p>

TGI_Fernando: <p>Did you install Ubuntu 20.04 on a virtual machine?</p>

Diego_Masotti: <p>No, directly on the disk.</p>

Diego_Masotti: <p>Ok solved. Ubuntu comes with Nvidia driver 440.62.<br>
I downloaded the latest 440.82 and then cuda works</p>

TGI_Fernando: <p>That’s great. Did you install only PyTorch? What about Anaconda and other libraries?</p>
Is it possible to send entire input and target to GPU instead of at individual step?

rkp768: <p>When I try to put enitre TainingData/Labels to GPU, the run seems to abruptly end without any error at line <code>op = net(x)</code>. But when I put individual inputs and labels to GPU the run works perfectly. Can anyone explain why ?</p>
<p>Also, pytorch tutorial mentions “Remember that you will have to send the <strong>inputs and targets at every step</strong> to the GPU too:”. Why not entire input/target at the beginning, instead of at every step ?</p>
<pre><code class="lang-auto">#TainingData = torch.Tensor(TainingData).to("cuda:0")
#Labels = torch.Tensor(Labels).to("cuda:0")
for epoch in range(1000):
    for i in range(1, TainingData.shape[0]+1):
        x, y = TainingData[i-1].reshape(1,1,28,28), Labels[i-1]
        y = torch.tensor([y], dtype=torch.long).to("cuda:0")
        x = torch.Tensor(x).to("cuda:0")
        op = net(x)
        loss = criterion(op, y)
</code></pre>

KFrank: <p>Hello rkp!</p>
<aside class="quote no-group" data-username="rkp768" data-post="1" data-topic="78419" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rkp768/48/23334_2.png" class="avatar"> rkp768:</div>
<blockquote>
<p>When I try to put enitre TainingData/Labels to GPU, the run seems to abruptly end without any error at line <code>op = net(x)</code>. But when I put individual inputs and labels to GPU the run works perfectly. Can anyone explain why ?</p>
</blockquote>
</aside>
<p>My guess would be that you’re getting a gpu out-of-memory error,<br>
but, for some reason, not getting a clean error message.</p>
<aside class="quote no-group">
<blockquote>
<p>Also, pytorch tutorial mentions “Remember that you will have to send the <strong>inputs and targets at every step</strong> to the GPU too:”. Why not entire input/target at the beginning, instead of at every step ?</p>
</blockquote>
</aside>
<p>I have done exactly this (with a smallish problem that fits in<br>
the gpu).</p>
<p>I moved my entire training (and test) set, both inputs and labels,<br>
to the gpu at the beginning, and then indexed into them with<br>
randomly shuffled indices to “create” my batches.  This worked<br>
fine for me.</p>
<p>Good luck.</p>
<p>K. Frank</p>

rkp768: <p>Thanks Frank !! Will try as suggested.</p>
Training Linear Layers without gradient available

VinceVJ98: <p>Hey everyone, this might be a somewhat unusual problem but I’m really hoping you are able to help me:</p>
<p>I’m trying to simulate some abstract trajectories and want to use a linear neural network to help with that. The way my algorithm works, is that for each step along the path the network gets the current state as an input and generates a few values, that I use to determine the next step.<br>
When the entire simulation is finished some statistical properties of the entire run are calculated to summarize essential features of the generated trajectory in a single function. The final goal for the network is to generate the coefficients for each next step in such a way, that the statistics of the resulting trajectory mirrors a given function I use as the target value. Because of that I always need to generate an entire trajectory, before I can check how well the network performed.</p>
<p>My problem arises when trying to compute the gradient. Because the network is used in every timestep, the exact equation to calculate the next step from the networks output is highly complicated and the statistical summarizing is added in the end, I am not able to reconstruct the influence of network weights on the final result and can’t give the optimizer the gradient it needs to train the network.</p>
<p>Do you have any idea how to solve this? Maybe there is an unconventional way of determining the gradients if the loss-function is calculated from very complicated and abstract results not trivially connected to the networks output?</p>
Is it possible to view values of a at::Tensor in Visual codes debug variable view (linux)?

Relop: <p>Hi!</p>
<p>I am currently learning about libtorch, c++ and visual code and wonder if someone knows if it is possible nicely debug libtorch inside visual code?</p>
<p>Essentially what I am after is to view the values of an at::Tensor in Visual codes debug variable view similar to how a torch.tensor can be viewed using the python plugin. Currently when I try to debug a simple program linking libtorch.so a basic at::Tensor (x) looks like below in the editor:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/9/7/97216b9bd77215eb145204907383239502d5b0e0.png" alt="example" data-base62-sha1="lyXHrSm3reDWHGBLU6hUWdTyNZm" width="348" height="107"></p>

pascal.soveaux: <p>Hello,</p>
<p>Something like  <code>*(float_t*)x.data_ptr()@size</code> ?</p>
<p>Pascal</p>