akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Concatenating a tuple of tensors using Python eval()

thatgeeman: <p>Suppose a module returns a tuple of classes and offsets at <code>L</code> feature layers (something like an SSD) for an object detection problem:</p>
<pre><code class="lang-auto">pred0_raw = SomeModule(params, 0)(x)
pred1_raw = SomeModule(params, 1)(x)  
pred2_raw = SomeModule(params, 2)(x)
</code></pre>
<p>Here, <code>len(pred0_raw)</code> is 2, corresponding to a tuple of <code>(class, offset)</code> predictions at layer <code>L=0</code>. <code>class</code> and <code>offset</code> are tensors of some shape <code>[B, N, C]</code>. I would like to concatenate them so I do this:</p>
<pre><code class="lang-auto">L=3
classes = [eval(f"pred{i}_raw[0]") for i in range(L)]  # class preds at predL_raw[0]
classes = torch.cat(classes, -2)  # classes.shape = B, N*L, C
# similarly offsets.shape = B, N*L, 4
</code></pre>
<p>When I execute these statements in a Jupyter cell for experimenting, it seems to work but I get a <code>NameError: name 'pred0_raw' is not defined</code> when I put all of that in a <code>Module</code>. What am I doing wrong here? Any alternative suggestions?<br>
Thanks!</p>

thatgeeman: <p>Alright! so this was simply due to the fact that the list comprehension puts <code>eval</code> in a different scope altogether! It was not a pytorch specific issue: <a href="https://stackoverflow.com/questions/45194934/eval-fails-in-list-comprehension" class="inline-onebox" rel="noopener nofollow ugc">python - eval fails in list comprehension - Stack Overflow</a>!</p>
How to use `torch.nn.utils.rnn.pad_sequence(sequences, batch_first=False, padding_value=0.0)` with set of parameters as `padding_value`?

gebrahimi: <p>Assume I have <code>x = torch.rand(2,50)</code> and <code>y = torch.rand(3,50)</code> and I have a tensor of trainable parameters p with size 50.</p>
<p>How can I use <code>torch.nn.utils.rnn.pad_sequence</code> or any other torch function to be able to create a <code>2*3*50</code> tensor where the padding_value is the tensor p?</p>
<p><code>torch.nn.utils.rnn.pad_sequence([x,y], batch_first=False, padding_value=p)</code></p>
<p>This throws an error since padding_value should be a number and not a tensor of variables.</p>
About tensor.grad

Yangmin: <p>Hi, I have a really simple question.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/3/63bc39114956d3827ad59df316ede30b9d8f0ab9.png" alt="image" data-base62-sha1="eeixbLVp24mxkMxYdd5YKIEdxJT" width="349" height="457"></p>
<p>In the above code, I understand why tensor b’s requires_grad and is_leaf changes after adding tensor a to b.<br>
But I don’t understand why b.grad is None and the a.grad value is<br>
a.grad == tensor([[1., 1.],<br>
[1., 1.]])</p>
<p>Could you give some explanation about this?</p>

albanD: <p>Hi,</p>
<p>b.grad is None because it is not a leaf. You can call <code>.retain_grad()</code> on it (after the inplace update) if you want the .grad field to be populated anyways.</p>
Correct way to implement Sampler

Mriganka_Nath: <p>I am working in a regression model, the data distribution is quite a multimodal type. So I wanted to make a sampler that can for target values greater than, say x(in code below, x =8) , is found at least 1% of all the dataset. Basically oversampling.<br>
But because of this my models are greatly overfitting.</p>
<pre><code class="lang-auto">class My_sampler(Sampler):
    def __init__(self ,dataset , pct = 0.1):
        self.df = dataset.df.Target
        self.pct = pct
    def __len__(self):
        return len(self.df)
    def __iter__(self):
        greater_idx = np.where(self.df &gt; 8)[0] 
        rest_idx = np.where(self.df &lt;= 8)[0]
        greater = np.random.choice(greater_idx , int(self.pct*len(self.df)) )
        rest = np.random.choice(rest_idx , int((1-self.pct)*len(self.df))+1 , replace = False)
        idxs = np.hstack([greater ,rest ])
        np.random.shuffle(idxs)
        idxs = idxs[:len(self.df)]
        return iter(idxs)
our_sampler = My_sampler(dataset)
loader = DataLoader(dataset , sampler=our_sampler , batch_size =8 , drop_last = True)
</code></pre>
<p>I don’t know why it’s overfitting, maybe I am not making the Sampler class properly.</p>
<p>Also is it possible that if I declare my sampler class like this, the same example is being used every batch, and in an epoch, every image is not being used as input?</p>
[TorchScript] Class with reference to instance of own Class

jellybelly: <p>Hi,<br>
currently I have to convert a model, and struggle to script a class that contains a <strong>parent</strong> member of the same class type.<br>
As</p>
<pre><code class="lang-auto">@torch.jit.script
class Tree(object):
    def __init__(self):
        self.parent:List[Tree]=[]
</code></pre>
<p>results in</p>
<pre><code class="lang-auto">RuntimeError: 
Assignment to attribute 'parent' cannot be of a type that contains class '__torch__.Tree '.
Classes that recursively contain instances of themselves are not yet supported:
</code></pre>
<p>I don’t think, this issue has already been tackled in TorchScript, so<br>
<strong>can someone please give me a hint, how to work around this problem?</strong></p>
<p>So far, I have tried:</p>
<pre><code class="lang-auto">class Model(Module):
  def __init__(self, parent:Any)-&gt;None:
    super().__init__()
    self.parent = [parent]

  def do_something(self, x:Tensor)-&gt;Tensor: return x

  def forward(self, x:Tensor)-&gt;Tensor:
    src = self
    if torch.jit.isinstance(self.parent[0], Model):
      src=self.parent[0]
      print('using parent')
    return src.do_something(x)

p = Model(None)
m = Model(p)
m(torch.rand(1,1))
xxx = torch.jit.script(m)
</code></pre>
<p>but not really successful…  <code>'Model' has no attribute 'parent'</code></p>
Create New Metric

Giulia_Carvalhal: <p>Hey community,</p>
<p>I am working on a segmentation model, and some objects have a lot of details. Because of that, the model recognizes these objects but it can not segment the details perfectly. Then the recall and IoU fall a lot.</p>
<p>So I was wondering if I could create somehow a metric that gives me 100% recall if the model segment correctly at least 70% of the object label, for example.</p>
<p>Best Regards,<br>
Giulia</p>

ItamarRocha: <p>I have a similar problem. I hope this question gets answered soon.</p>
Fastest way to calculate matthews correlation coefficient

M2M: <p>Hello everyone,</p>
<p>I am working with semantic segmentation where I end up comparing very large tensors of labels against the predictions (each containing more than 5.6 million pixels). At the moment, I just convert the labels and predictions into python lists and use sklearn to calculate Matthews Correlation Coefficient but this is super slow. I would really appreciate any support to speed up this. Pytorch is fast running epochs but when I try to use sklearn, it gets super slow.</p>
<p>The portion of my code</p>
<pre><code class="lang-auto">y_true = []
y_pred = []

with torch.no_grad():
 for data in data_loader:
  images, labels = data
  images = images.to(device)
  y_true = y_true + torch.flatten(labels).tolist() # Tensor is still in CPU to avoid copying it to CPU list y_true
  labels = labels.to(device)

  outputs = net(images)
  _, predicted = torch.max(outputs.data, 1)

 predicted = predicted.to('cpu')
 y_pred = y_pred + torch.flatten(predicted).tolist()

mcc = matthews_corrcoef(y_true, y_pred) # The line that makes the whole code slow

</code></pre>
<p>Any help regarding this will be highly appreciated.</p>
Inconsistency between loss-functions input Types

Jakob: <p>When using the <code>nn.BCEloss</code> the <code>target</code>  has to be a tensor with dtype <code>Float</code> whereas <code>nn.CrossEntropyLoss</code> it has to be <code>Long</code>.</p>
<p>Why is that so? Why not stick to either Long or Float?</p>
<p>It make it difficult to create a wrapper since we cannot cast the input/target to a specific (static) dtype, thus one either have to make a <code>try/catch</code> statement, do the conversion based on an error-message, which of calls for a lot of unnecessary checks</p>

KFrank: <p>Hi Jakob!</p>
<aside class="quote no-group" data-username="Jakob" data-post="1" data-topic="138942" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/71c47a/48.png" class="avatar"> Jakob:</div>
<blockquote>
<p>Why not stick to either Long or Float?</p>
</blockquote>
</aside>
<p>Historically, pytorch’s <code>CrossEntropyLoss</code> was restricted to integer<br>
class labels for its <code>target</code> and did not support probabilistic “soft”<br>
labels.  These are naturally integers, hence <code>Long</code>.  In contrast,<br>
<code>BCELoss</code> (and its to-be-preferred sibling, <code>BCEWithLogitsLoss</code>)<br>
support (and require) probabilities for <code>target</code>, hence <code>Float</code> and<br>
<code>Double</code>.</p>
<p>However, as of pytorch version 1.10.0, <code>CrossEntropyLoss</code> supports<br>
both categorical (integer) and probabilistic <code>target</code> values, so this<br>
inconsistency should be gone now.</p>
<p>Best.</p>
<p>K. Frank</p>
How to create torch::tensor constant in c++?

MartinZhang: <p>there is  a question about how to create a torch::Tensor constant in cpp?</p>
<p>I have some data table, like this:</p>
<pre><code class="lang-auto">const int a [] = {123, 456, 789};
</code></pre>
<p>but now , I want to translate the type of <code>a</code>  into torch::Tensor, such as</p>
<pre><code class="lang-auto">#include &lt;torch/extension.h&gt;

torch::Tensor a = {123, 456, 789};
</code></pre>
<p>in this link <a href="https://pytorch.org/cppdocs/api/classat_1_1_tensor.html?highlight=torch%20tensor" rel="noopener nofollow ugc">https://pytorch.org/cppdocs/api/classat_1_1_tensor.html?highlight=torch%20tensor</a></p>
<p>I’m not find a good way to use it.</p>
<p>please help me, thanks.</p>

ptrblck: <p><code>from_blob</code> should work:</p>
<pre><code class="lang-python">  float data[] = {1, 2, 3, 4, 5, 6};

  auto f = from_blob(data, {1, 2, 3});
</code></pre>
<p>Make sure the <code>data</code> is alive as long as you are using <code>f</code> or <code>clone()</code> the data.</p>

MartinZhang: <p>It’s awesome, Thanks. <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=10" title=":grinning:" class="emoji" alt=":grinning:"></p>
Training get stuck at some iteration step

GeoffreyChen777: <p>I’m using PyTorchv1.1.0 and DistributedDataParallel to train some models.</p>
<p>The training process will get stuck at some constant steps.</p>
<p>I noticed some wired things:</p>
<ol>
<li>The Memory usage keeps freeze.</li>
<li>GPU usage increases to 100%. But the temperature of GPU isn’t very high.</li>
<li>The CPU usage of 4 main progress is 100%</li>
</ol>
<p>I think it is not a code bug. Because I use some different models and they both get stuck at some iteration step.</p>

Mr.Z: <p>I met this problem before, then I installed pytorch nightly (version: 1.2.0.dev20190628) and the problem seems been solved.</p>

ukemamaster: <p><a class="mention" href="/u/geoffreychen777">@GeoffreyChen777</a> I have exactly the same problem. Did you solve your problem? If yes, could you please share your solution?<br>
I tried pytorch nightly as suggested by <a class="mention" href="/u/mr.z">@Mr.Z</a> , but it didn’t solve my problem.</p>

GeoffreyChen777: <p>Hi, I remember that no idea about why it happens when I use PyTorch 1.1.0.</p>
<p>In the past year, I never met this with newer version of PyTorch. Maybe you can try the newest.</p>

ukemamaster: <p>I tried with pytorch 1.6.0, 1.7.0, 1.8.0, and 1.9.0. All have the same problem.</p>

breakds: <p>Hi <a class="mention" href="/u/ukemamaster">@ukemamaster</a> , did you find a solution now? I think I am running into the same problem.</p>

ukemamaster: <p><a class="mention" href="/u/breakds">@breakds</a> Somehow yes. In my case it was I/O (data loading) problem. I was reading my training data from HDD on the fly (reading raw audio files, plus data augmentation/pre-processing) which was the very slow. When i put my dataset on SSD, everything was fine.</p>
Torch.autograd.functional.hvp does not work on a model in the evaluation mode

Naoya: <pre><code class="lang-auto">import torch
from torch.autograd.functional import hvp
from torchvision import models

model = models.resnet18()
model.eval() # model.train()

x = torch.randn((1, 3, 224, 224))
hvp(lambda _x: model(_x).max(), x, x, strict=True)
</code></pre>
<p>The above code returns the following message:</p>
<blockquote>
<p>RuntimeError: The hessian of the user-provided function is independent of entry 0 in the grad_jacobian. This is not allowed in strict mode as it prevents from using the double backward trick to replace forward mode AD.</p>
</blockquote>
<p>If I turn on the training mode, i.e., change to <em>“model.train()”</em> from <em>model.eval()</em>, the function hvp can work.<br>
However, I suspect that this is not good for a model including “Dropout” or “BatchNorm”.</p>
<p>Can anyone help me in resolving this issue?</p>
<p>Development environment;</p>
<ul>
<li>Ubuntu 20.04</li>
<li>Python 3.9.5</li>
<li>torch 1.10.0</li>
<li>torchvision 0.11.1</li>
</ul>

Naoya: <pre><code class="lang-auto">model.eval()
# list(model.children())[1] is the first BatchNorm2D layer
list(model.children())[1].training = True 
</code></pre>
<p>I could run hvp by the above modification.<br>
However, it looks still to be an issue and strange.</p>
Get NaN in nn.Softmax when the input is created by -np.inf

Rancho_Xia: <p>As the title suggests, I created a tensor by <code>a = torch.zeros((3, 4)).fill_(-np.inf)</code>. Theretically, every element of a is a super small negative value, and <code>nn.Softmax(a)</code> should produce near zero output. However, the output is NaN.Is there something I missed or misunderstood? Any help is appreciated!<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/a/e/ae8dbe49da79311eac0808cfccf4429953aa155a.png" alt="QQ截图20211208232008" data-base62-sha1="oUaLkVuYEQUgqX1pfBsv0B4TiUq" width="566" height="300"></p>

Rancho_Xia: <p>Currently, I’m training a Transformer, and the loss is NaN after the const value in the mask is set to -np.inf, but it works well with the const being -1e20</p>

mMagmer: <p>i think using <code>torch.nn.LogSoftmax</code> might work for you.<br>
LogSoftmax is numerically more stable, but in your test code still return <code>nan</code>.</p>

Azerus: <p>Hello,</p>
<p>From my understanding of Softmax, it’s quite inconsistent to have all your logits = -inf, you should have at least one logit which value is superior to -inf. Ex:</p>
<pre><code class="lang-auto">&gt;&gt;&gt;
&gt;&gt;&gt; a = torch.zeros(4)
&gt;&gt;&gt; a[0:3] = float('-inf')
&gt;&gt;&gt; a
tensor([-inf, -inf, -inf, 0.])
&gt;&gt;&gt; torch.nn.functional.softmax(a)
__main__:1: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.
tensor([0., 0., 0., 1.])
&gt;&gt;&gt; 
&gt;&gt;&gt; 
&gt;&gt;&gt; a[0:4] = float('-inf')
&gt;&gt;&gt; torch.nn.functional.softmax(a)
tensor([nan, nan, nan, nan])

</code></pre>
<p>Also, about your statement:</p>
<blockquote>
<p>Theretically, every element of a is a super small negative value, and <code>nn.Softmax(a)</code> should produce near zero output.</p>
</blockquote>
<p>It is true that the probability value in the output of a softmax for a logit that tend to -inf should tend to 0. But your output probability distribution cannot all be close to 0 evrywhere, as the sum of all probabilities should sum to 1.</p>

Rancho_Xia: <p>Thanks for your reply. Sadly it doesn’t work either. This question doesn’t bother me to write a Transformer because I set the const in the mask to -1e20 and it works pretty well. I just want to know why <code>-np.inf</code> fails. According to some reference codes, it seems that -np.inf once worked. I guess there’s some version change which leads to this pitfall. weird thing.</p>
Need Suggestions for upgrading older version pytorch

umfengxx: <p>Hi guys, my model can only work with pytorch &lt;=1.1.0, unfortunately pytorch 1.1.0 can’t work with CUDA 11, which is needed in new GPU card like Nvidia A100 and 30xx series.</p>
<p>When I tried to use pytorch 1.7 + CUDA 11, my trained model will produce strange noise in output images.  Model trained on pytorch 1.1.0 is OK, models trained on pytorch 1.1.0 can work in a forward loop in pytorch 1.7+CUDA11 environment.</p>
<p>My question is,</p>
<ol>
<li>If I want to upgrade to pytorch 1.7, is there any guideline how to upgrade from pytorch 1.1.0 to pytorch 1.7, since there are many upgraded functions between versions. I don’t know which function I should pay attention to.</li>
<li>If I want to use pytorch 1.1.0 with CUDA 11 in new GPU card, is possible, and how to?</li>
</ol>
<p>Thanks for the attention~</p>

ptrblck: <ol>
<li>
<p>You could skim through the release notes for each released PyTorch version and check which changes might affect your training. I would probably start with backwards compatibility breaking changes and see, if fixing these might help.</p>
</li>
<li>
<p>You would have to search for all CUDA 11.x related PRs and try to backport them to the older PyTorch release, which might be cumbersome depending how old the PyTorch release is (and might not be trivial at all).</p>
</li>
</ol>

umfengxx: <p>For qestion 1, there might be a huge amount of work to do. I’ll think about it. Maybe some debugging skills can simplify this work? I totally agree with you to start with backwards part. <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=10" title=":smiley:" class="emoji" alt=":smiley:"><br>
For qestion 2, in fact I just need pytorch 1.1.0,  is it possible to build from source for this version using CUDA 11?<br>
Many thanks for the helpful reply.</p>

ptrblck: <p>I haven’t tried to build PyTorch <code>1.1.0</code> with any CUDA 11 release, but since <code>1.1.0</code> was released in April 2019, I’m pretty sure it won’t work without some backports.</p>
How to create Dataset out of Matlab Array?

FreSch48: <p>Hi guys,</p>
<p>I am absolutely new here and therefore have little or no experience with PyTorch.<br>
I would like someone to be able to tell me the best way to use a data array from Matlab as a data set, preferably with a normalization of the data, e.g. in the range [0, 1].</p>
<p>Here a little more detail about my data:<br>
I have a two-dimensional array, with the rows corresponding to frames from several spectrograms. The size of the array is 60000 x 1025, i.e. 60000 frames with 1025 frequency bins. Since I use an auto encoder, my input data is also my target data.</p>
<p>I already know that I can use mat73.loadmat (path) to read my Matlab data as an array in Python. Now the only question is how can I create a normalized dataset from this?</p>
<p>I hope you guys can help me.<br>
Thanks in advance</p>

mMagmer: <p>if you are loading your data to a numpy array, do as follow:</p>
<pre><code class="lang-auto">dataset = torch.from_numpy(data)
</code></pre>
<p>and if your data is loading to python list:</p>
<pre><code class="lang-auto">dataset = torch.tensor(data)
</code></pre>
<p>for data normalization:</p>
<pre><code class="lang-auto">dMin  = (dataset.min(0,keepdim=True))[0]
dMax = (dataset.max(0,keepdim=True))[0]
NDataset = (dataset - dMin)/(dMax - dMin)
</code></pre>
<p>then you need to define data loader:</p>
<pre><code class="lang-auto">dl = torch.utils.data.DataLoader(NDataset,batch_size=100,shuffle=True)
</code></pre>
<pre><code class="lang-auto">for i in range(epoch):
    for x in dl:
         #do something with it.
</code></pre>

FreSch48: <p>Thanks for your help.</p>
Multiple dataloaders

R_S: <p>hello, i am using this command to get the batches from 2 different datat loaders.<br>
Is this correct? Thank you</p>
<p>first method</p>
<pre><code class="lang-auto">for iter_id, item1 in enumerate(tqdm(train_loader)):
    iter_id2, item2 = enumerate(target_loader).__next__()
    image, labels = item1
    image2 = item2

    print(image.shape, image.dtype)
    image = np.transpose(image, (0, 2, 3, 1))
    plt.imshow(image[2])
    plt.show()

    print(image2.shape, image2.dtype)
    image2 = np.transpose(image2, (0, 2, 3, 1))
    plt.imshow(image2[2])
    plt.show()
</code></pre>
<p>second method</p>
<pre><code class="lang-auto">for (item1, item2) in zip(cycle(train_loader), target_loader):
    image, labels = item1
    image2 = item2

    print(image.shape, image.dtype)
    image = np.transpose(image, (0, 2, 3, 1))
    plt.imshow(image[2])
    plt.show()

    print(image2.shape, image2.dtype)
    image2 = np.transpose(image2, (0, 2, 3, 1))
    plt.imshow(image2[2])
    plt.show()
</code></pre>

my3bikaht: <p>This code is confusing. Imagine if you forget to disable shuffle in one of the dataloaders or use multiple workers.</p>
<p>Simpler way here is to define your custom dataset with all the data. Also you can move all preprocessing outside of the loop either with dataset’s init/get_item or with collate_fn.</p>
How to get a module through its key in state_dict

Forceless: <p>What I expect:</p>
<ol>
<li>get model’s state_dict and its keys</li>
<li>get submodules of model through these keys</li>
<li>register forward hook on these modules</li>
</ol>
<p>Why I cannot use model.children:<br>
I have select modules’ weight_dict through key word filter and it’s hard to say what kind of module will be processed after</p>
Module 'torchvision.models' has no attribute 'efficientnet_b0'

Ammara: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/3/93770c62ac2a7a4721f50fca6c712177a73c8436.png" data-download-href="https://discuss.pytorch.org/uploads/default/93770c62ac2a7a4721f50fca6c712177a73c8436" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/3/93770c62ac2a7a4721f50fca6c712177a73c8436.png" alt="image" data-base62-sha1="l2xfnPyFR1SbM8VDqxxifj7XLv0" width="690" height="134" data-dominant-color="292929"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">915×178 10.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I am getting the following error when trying to use the efficientNet architectures in torchvision models.</p>

my3bikaht: <p>It’s in 0.11.0 release</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/pytorch/vision/releases">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/pytorch/vision/releases" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/345;"><img src="https://opengraph.githubassets.com/df37e6447f47d321f34954c780fd5a7b5fcff690cdd4b76e7826574ad399c664/pytorch/vision" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/pytorch/vision/releases" target="_blank" rel="noopener nofollow ugc">Releases · pytorch/vision</a></h3>

  <p>Datasets, Transforms and Models specific to Computer Vision - pytorch/vision</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


Ammara: <p>Thankyou.<br>
The issue was with the torchvision version.</p>
Torch.save()/can not save the whole tensor if the tensor takes more than 10 GB storage

Johnson: <p>What should I do to deal with this problem?</p>
<p>Thank you very much in advance!</p>

ptrblck: <p>I cannot reproduce the issue in a recent master build and am able to save a tensor of e.g. 16GB using:</p>
<pre><code class="lang-python">&gt;&gt;&gt; import torch
&gt;&gt;&gt; x = torch.randn(int(4*1024**3), device='cuda')
&gt;&gt;&gt; print(torch.cuda.memory_allocated()/1024**3)
16.0
&gt;&gt;&gt; torch.save(x, 'tmp.pt')
&gt;&gt;&gt; y = torch.load('tmp.pt')
&gt;&gt;&gt; print(y.shape)
torch.Size([4294967296])
</code></pre>

Johnson: <p>I used the same operation as you, but an error occurred when I load the built tensor.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/8/38b1d874a5838e28a14a81bdd5017b3b7b6d31e7.png" data-download-href="https://discuss.pytorch.org/uploads/default/38b1d874a5838e28a14a81bdd5017b3b7b6d31e7" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/8/38b1d874a5838e28a14a81bdd5017b3b7b6d31e7.png" alt="image" data-base62-sha1="85xLMk2U3SEXJC9n40iUbiaiDMr" width="690" height="185" data-dominant-color="302F32"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1403×378 27.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>I’m not sure if this issue might be Windows-specific, but could you update PyTorch to the latest nightly release and check, if this might have been an already known and fixed issue, please?</p>

Johnson: <p>I will try according to your suggestion. Thank you for your help!</p>
How to interpret the following loss graph of CGAN?

Sumera_Rounaq: <p>I am training CGAN for generating Xray images.</p>
<p><strong>My training initializations are as follows:</strong></p>
<pre><code class="lang-auto">criterion = nn.BCEWithLogitsLoss()
training samples = 1200
n_epochs = 600
z_dim = 100
batch_size = 50
lr_gen = 0.0002
lr_disc = 0.0001
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/a/5ab51401bd71d6d433b668c2097a6fda1b7bc1fe.png" alt="CGAN-loss graph" data-base62-sha1="cWqVGO1I79226wuvKRJiu7qnh0O" width="368" height="248"></p>
<p>This is the loss graph after 500 epochs. I am not able to interpret whether training is working fine or not. What does this generator’s and discriminator’s graph conclude?</p>
<p>And following is the output of fake images for 554th epoch.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/f/4f4a097daddeec77b5767fa18a8bed05902017b5.png" data-download-href="https://discuss.pytorch.org/uploads/default/4f4a097daddeec77b5767fa18a8bed05902017b5" title="fake_images-0554"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/f/4f4a097daddeec77b5767fa18a8bed05902017b5_2_690x345.png" alt="fake_images-0554" data-base62-sha1="bjqljOCVdxJ4O0EVrGjvVUDpuo5" width="690" height="345" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/f/4f4a097daddeec77b5767fa18a8bed05902017b5_2_690x345.png, https://discuss.pytorch.org/uploads/default/original/3X/4/f/4f4a097daddeec77b5767fa18a8bed05902017b5.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/f/4f4a097daddeec77b5767fa18a8bed05902017b5.png 2x" data-dominant-color="5C5C5C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">fake_images-0554</span><span class="informations">1022×512 238 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>The losses would indicate that the discriminator is successful in differentiating fake from real examples (thus the discriminator loss is low) while the generator has trouble creating images which the discriminator would classify as “real” images.</p>
CNN Input size to fully connected layer

vbjohnsen: <p>Hi, trying to make a CNN work but i am getting errors regarding the input to the fc. I understand that it is the input that is wrong but i cant seem to get it right any help would be much appreciated.</p>
<p>Here is the network</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
      super(SecondCNN, self).__init__()
      self.conv1 = nn.Conv2d(3, 6, 5)
      self.pool = nn.MaxPool2d(1, 1)
      self.conv2 = nn.Conv2d(6, 16, 5)
      self.conv3 = nn.Conv2d(16, 32, 5)
      self.fc1 = nn.Linear(32 * 20 * 20 , 120)
      self.fc2 = nn.Linear(120, 84)
      self.fc3 = nn.Linear(84, 10)

    def forward(self, x):

      x = self.pool(F.relu(self.conv1(x)))
      print(x.shape)
      x = self.pool(F.relu(self.conv2(x)))
      print(x.shape)
      x = self.pool(F.relu(self.conv3(x)))
      print(x.shape)
      x = F.relu(self.fc1(x))
      print(x.shape)
      x = F.relu(self.fc2(x))
      x = self.fc3(x)
      return x
</code></pre>
<p>Here is the size of the first input to the network: torch.Size([4, 3, 32, 32])</p>
<p>And as you can see im printing out the shape along the way as im trying to debug this, i only get the first three as it throws the runtime error before the fc.</p>
<p>Here are the shapes:</p>
<pre><code class="lang-auto">torch.Size([4, 6, 28, 28])
torch.Size([4, 16, 24, 24])
torch.Size([4, 32, 20, 20])
</code></pre>

ptrblck: <p>Your model architecture looks good and also the debugging steps printing the shapes are great.<br>
The issue is raised because you are not flattening the activation tensor via <code>x = x.view(x.size(0), -1)</code> before passing it to <code>self.fc1</code>. Add this line of code and it should work.</p>
CTC Loss - Add Weights

Vishak_Raj: <p>Hello,</p>
<p>I am learning CTC Loss from <a href="https://pytorch.org/docs/stable/generated/torch.nn.CTCLoss.html" class="inline-onebox" rel="noopener nofollow ugc">CTCLoss — PyTorch 1.10.0 documentation</a>,</p>
<p>In CTC Loss, how should we add weights like in <code>torch.nn.CrossEntropyLoss</code> to handle unbalance dataset?</p>
<p>Thanks</p>
Shap values, RuntimeError: grad can be implicitly created only for scalar outputs'''

Saida2020: <p>after i trained the model and try to use shap explainer this line rasie an error<br>
‘’‘shap_values = e.shap_values(sequences_to_explain) ‘’’</p>
<p>‘’‘raise RuntimeError(“grad can be implicitly created only for scalar outputs”)<br>
RuntimeError: grad can be implicitly created only for scalar outputs’’’</p>
<p>I already looked at other posts that said it related to loss.backward but the answers did not work for me.</p>
<p>Any suggestions</p>

ptrblck: <p>I don’t know what <code>shap_values</code> is calling internally, but the error message indicates that <code>backward()</code> is called on a tensor, which contains more than a single element (i.e. isn’t a scalar tensor).<br>
In these cases you would either need to reduce the tensor to a scalar (e.g. via <code>torch.mean</code>) or would need to provide a gradient tensor to <code>backward</code> via <code>tensor.backward(grad)</code>, where <code>grad</code> has the same shape as <code>tensor</code>.</p>

Saida2020: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>  Thank you so much for you reply, I tried that answer but I got same error<br>
Here is the full error<br>
‘’’  File “/mnt/train_model.py”, line 247, in <br>
shap_values = e.shap_values(sequence)<br>
File “/home/Guest/.local/lib/python3.6/site-packages/shap/explainers/_deep/<strong>init</strong>.py”, line 124, in shap_values<br>
return self.explainer.shap_values(X, ranked_outputs, output_rank_order, check_additivity=check_additivity)<br>
File “/home/Guest/.local/lib/python3.6/site-packages/shap/explainers/_deep/deep_pytorch.py”, line 185, in shap_values<br>
sample_phis = self.gradient(feature_ind, joint_x)<br>
File "/home/Guest/.local/lib/python3.6/site-packages/shap/explainers/<em>deep/deep_pytorch.py", line 123, in gradient<br>
allow_unused=True)[0]<br>
File “/home//Guest/.local/lib/python3.6/site-packages/torch/autograd/<strong>init</strong>.py”, line 229, in grad<br>
grad_outputs</em> = <em>make_grads(outputs, grad_outputs</em>)<br>
File “/home//Guest/.local/lib/python3.6/site-packages/torch/autograd/<strong>init</strong>.py”, line 51, in _make_grads<br>
raise RuntimeError(“grad can be implicitly created only for scalar outputs”)<br>
RuntimeError: grad can be implicitly created only for scalar outputs ‘’’’</p>

ptrblck: <p>It’s hard to tell why the code is failing as it seems that <code>self.gradient</code> is called internally in:</p>
<pre><code class="lang-python">File “/home/Guest/.local/lib/python3.6/site-packages/shap/explainers/_deep/deep_pytorch.py”, line 185, 
</code></pre>
<p>which points to the used <code>shap</code> package. Could you cross-post the issue in their GitHub so that the code owners could also check if this is a known issue or is your usage of the package might be wrong?</p>
ValueError: 'label' is not in list

adityawelly: <p>I was doing sentiment analysis,</p>
<pre><code class="lang-auto">import torch
from torchtext import data
from torchtext import datasets
import random
import numpy as np

SEED = 1234

torch.manual_seed(SEED)
torch.backends.cudnn.deterministic = True

TWEET = data.Field(tokenize = 'spacy', batch_first = True)
USERNAME = data.Field()
LABEL = data.LabelField(dtype = torch.float)

import pandas as pd
import os

train_data = pd.read_csv(r"D:\KONTOL\train.csv")

fields = {'tweet_text': ('tweet', TWEET), 'username': ('username', USERNAME), 'label': ('label',LABEL)}

train_data, valid_data, test_data = data.TabularDataset.splits(
                                        path = 'D:\KONTOL',
                                        train = 'train.csv',
                                        validation = 'valid.csv',
                                        test = 'test.csv',
                                        format = 'csv',
                                        fields = fields,
                                        skip_header = False
)
</code></pre>
<p>from the data that I scraped, I added a label column and its contents (positive and negative) to my data.  but when I call the label column, it doesn’t read and finds the following error:</p>
<pre><code class="lang-auto">ValueError: 'label' is not in list
</code></pre>
<p>is there a way to solve it? any help is extremly appreciated</p>

Tanvi_Khosla: <p>hey did you get the solution to this problem i have the same error</p>
Forecast future points in timeseries data

NullPointer: <p>How does a model forecast future data without those points in time? I trained my model, and validated it, but I’m struggling with predictions.</p>
<pre><code class="lang-auto">def predict_future(seconds: int, future_data, df):
    all_preds = []
    model.eval()
    seq = torch.from_numpy(future_data[:batch_size]).float()

    with torch.no_grad():
        preds = model(seq).detach().to('cpu').numpy()

        preds = np.squeeze(preds[:seconds]).shape

        preds = undo_normalize_0_to_1(preds, df.min, df.max)

        all_preds.append(preds)
        print(preds)

        return all_preds
</code></pre>
<p>This is my code, but it doesn’t predict future points, it only makes predictions on “seconds” number of points in the array future_data. I was thinking maybe I need to shift my data so that each point corresponds to the future point, but I wasn’t sure if that was a legitimate approach to this.</p>
Pytorch matmul is inconsistent on GPU

Kenny_Mccormic: <pre><code class="lang-auto">xs = torch.randn(4, 18, 2, device='cuda')
ys = torch.randn(2, 2, device='cuda')

print((xs @ ys)[0, 0, 0])
print((xs[0, 0].unsqueeze(0) @ ys)[0, 0])
print(torch.matmul(xs, ys)[0, 0, 0])
</code></pre>
<p>the first and second output can be slightly different (up to 1e-4);<br>
the device is 3090 and pytorch version is 1.7.1;<br>
the inconsistency has not been observed on cpu.</p>

AlphaBetaGamma96: <p>It’d be best to check with a dev, but it’s possibly due to the fact that 30 series cards will default to using TensorFloat32 whereas your CPU will default to Float32. (See <a href="https://pytorch.org/docs/stable/notes/cuda.html#tf32-on-ampere" rel="noopener nofollow ugc">here</a> and <a href="https://blogs.nvidia.com/blog/2020/05/14/tensorfloat-32-precision-format/" rel="noopener nofollow ugc">here</a> for more detail)</p>
<p>TensorFloat32 has the same range of Float32 but has the precision of Float16, so you’re probably seeing a round-off error with the small error of 1e-4. You can set this behaviour to False via <code>torch.backends.cuda.matmul.allow_tf32</code> and <code>torch.backends.cudnn.allow_tf32</code>. More detail is here: <a href="https://pytorch.org/docs/stable/notes/cuda.html#tf32-on-ampere" class="inline-onebox" rel="noopener nofollow ugc">CUDA semantics — PyTorch 1.10.0 documentation</a></p>

my3bikaht: <p>Here’s my output for your code:</p>
<pre><code class="lang-auto">tensor(0.2188241482, device='cuda:0')
tensor(0.2188241482, device='cuda:0')
tensor(0.2188241482, device='cuda:0')
</code></pre>
<p>3080ti, torch 1.10.0</p>

Kenny_Mccormic: <p>That’s weird, here are some of my outputs:</p>
<pre><code class="lang-auto">tensor(-0.9371, device='cuda:0')
tensor(-0.9373, device='cuda:0')
tensor(-0.9371, device='cuda:0')
</code></pre>
<pre><code class="lang-auto">tensor(-0.1670, device='cuda:0')
tensor(-0.1665, device='cuda:0')
tensor(-0.1670, device='cuda:0'
</code></pre>
<pre><code class="lang-auto">tensor(0.5239, device='cuda:0')
tensor(0.5238, device='cuda:0')
tensor(0.5239, device='cuda:0')
</code></pre>

AlphaBetaGamma96: <p>You could try and force deterministic to be True? And see if your results are the same?</p>
Style transfer implementation: artefacts in content loss

juanigp: <p>Hello, as many others I am implementing style transfer from scratch. Since my generated images were not very convincing, I decided to first get the content loss part working, and just try to reconstruct an image from a randomly initialised tensor.</p>
<p>These are some results, and I cannot get rid of those green and violet pixel artefacts.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/d/cd5d6c32395429b53f7e864cf61202cc76b499d6.png" alt="download" data-base62-sha1="tiK2o6jnlRfekBm29oK8wcwmDyK" width="264" height="252"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/0/6/068aebfdf7e798fb66d7edb2cf97830748721b4d.png" alt="download (1)" data-base62-sha1="VSvdskxEyRrQEBRjVrJ2US8cNv" width="257" height="252"></p>
<p>While trying to understand how to solve this, I found this repo which makes super pretty images, and it can reconstruct content images from a random tensor without getting the artefacts I get.</p>
<p><a href="https://github.com/EugenHotaj/pytorch-generative/blob/master/pytorch_generative/colab_utils.py" rel="noopener nofollow ugc">https://github.com/EugenHotaj/pytorch-generative/blob/master/pytorch_generative/colab_utils.py</a></p>
<p>Just to clarify, I tried that code without the modifications they introduce (I used vgg16 instead of vgg19, i kept the maxpool layers and inplace relu, and tried different layers to get the content activations). Nevertheless the reconstructions from that repo seem to not have the same artefacts I get.</p>
<p>I would appreciate any help! This is a google colab with my code:<br>
<a href="https://colab.research.google.com/drive/1YD1_WvuTOJfRmeUwcvkHs6iv9UOX_nYa?usp=sharing" rel="noopener nofollow ugc">https://colab.research.google.com/drive/1YD1_WvuTOJfRmeUwcvkHs6iv9UOX_nYa?usp=sharing</a></p>

juanigp: <p>I get rid of the artefacts by clamping the pastiche tensor one more time after all the processing. I would like to think that this thread is useful to someone else since I struggled with this issue a lot (:</p>
How can I determine if a tensor has any named dimensions?

ProGamerGov: <p>I would like to have my code used the named dimensions unless no dimensions exists, but I’m not sure how I would determine this.</p>
<p>Example:</p>
<pre><code class="lang-auto">x1 = torch.ones(1,3,4,4)
x2 = torch.ones(1,3,4,4).refine_names("D0", "D1", "D2", "D3")

x1 = #No named dims
x2 = # Has named dims
</code></pre>

ProGamerGov: <p>I figured it out! Somehow I missed that tensors have a <code>.names</code> attribute that gives me the required information:</p>
<pre><code class="lang-auto">x1 = torch.ones(1,3,4,4)
x2 = torch.ones(1,3,4,4).refine_names("D0", "D1", "D2", "D3")

print(x1.names)
print(x2.names)

(None, None, None, None)
('D0', 'D1', 'D2', 'D3')
</code></pre>
Is there any method to read 4 channel transperant image in PyTorch?

vishalthengane: <p>I am working on an image-to-image translation GANs. I have a transparent 4 channel image. But I am getting on 3 channels with PIL and the transparent part is getting blacked out. How I can read the image in PyTorch. Is there any method available?</p>

zilong: <p>I think you should convert the image to the right format using PIL could be the solution:</p>
<pre><code class="lang-auto">from PIL import Image

img = Image.open(img_path)

# converts to 4 channels, where A represents the alpha/transparent channel
img.convert(RGBA)
</code></pre>
Pytorch version for torch.tile and torch.repeat

cbd: <p>I used “torch.tile” and “torch.repeat” in google colab and working fine. After printing “print(torch.<strong>version</strong>)” giving the version “1.10.0+cu111”.</p>
<p>In PC, After printing “print(torch.<strong>version</strong>)” gives the version " 1.7.1". For both the function,It shows the error “module torch has no attribute”</p>
<p>(1) Which version should i need to install to use torch.tile and torch.repeat?<br>
(2) Should i replace torch.tile with torch.repeat?</p>

zilong: <p>Installing torch version 1.10.0 on your PC should work</p>
Back Propagating the loss in joint training

AbraMunna: <p>I have a joint network where the data go through two separate subnetworks and get concatenated before sending through a third network. I am just wondering how can I back propagate two losses such a way that the optimizer will update net1’s parameter by loss1 and net2’s parameters by loss2</p>
<p>Models could be these,</p>
<pre><code class="lang-auto">class net1(nn.Module):
    def __init__(self, input_size, hidden_size, num_classes, p=0.0):
        super().__init__()
        self.embeddings = nn.Sequential(
            nn.Linear(input_size, hidden_size),
            nn.ReLU()
        )
    def forward(self, x):
        return self.embeddings(x)

class net2(nn.Module):
    def __init__(self, input_size, hidden_size, num_classes, p=0.0):
        super().__init__()
        self.embeddings = nn.Sequential(
            nn.Linear(input_size, hidden_size),
            nn.ReLU()
        )
    def forward(self, x):
        return self.embeddings(x)
</code></pre>
<p>Training loop:</p>
<pre><code class="lang-auto">loss1 = net1(data)
loss2 = net2(data)
loss = loss1 + loss2
loss.backward()
optimizer.step()
</code></pre>
<p>I am just wondering, would that make sure net1 would be updated by ‘loss1’ and net2 by ‘loss2’?<br>
or it will get a common updates from ‘loss’</p>
<p>Thanks in advance</p>

Azerus: <p>Hi,</p>
<p>Looking at your training loop, yes net1 will be updated by ‘loss1’ and net2 by ‘loss2’, as derivative of a sum is the sum of the derivatives. You can verify it by comparing the gradient of your models when doing backward on the total loss versus doing backward on the individual losses, ex:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; # Create dummy parameters
&gt;&gt;&gt; param_1 = torch.nn.Parameter(torch.ones(1))
&gt;&gt;&gt; param_2 = torch.nn.Parameter(torch.ones(1))
&gt;&gt;&gt; 
&gt;&gt;&gt; # Create dummy losses
&gt;&gt;&gt; loss1 = param_1*5
&gt;&gt;&gt; loss2 = 2**param_2
&gt;&gt;&gt; 
&gt;&gt;&gt; # Backward separatly on loss1 and loss2
&gt;&gt;&gt; loss1.backward()
&gt;&gt;&gt; loss2.backward()
&gt;&gt;&gt; 
&gt;&gt;&gt; # Check gradient of param_1 and 2
&gt;&gt;&gt; param_1.grad
tensor([5.])
&gt;&gt;&gt; param_2.grad
tensor([1.3863])
&gt;&gt;&gt; 
&gt;&gt;&gt; 
&gt;&gt;&gt; # Let's redo the experiment with the same dummy parameters and losses
&gt;&gt;&gt; param_1 = torch.nn.Parameter(torch.ones(1))
&gt;&gt;&gt; param_2 = torch.nn.Parameter(torch.ones(1))
&gt;&gt;&gt; loss1 = param_1*5
&gt;&gt;&gt; loss2 = 2**param_2
&gt;&gt;&gt; 
&gt;&gt;&gt; # But this time let's backward on their sum
&gt;&gt;&gt; loss = loss1 + loss2
&gt;&gt;&gt; 
&gt;&gt;&gt; # Let's check that the gradient are the same as earlier:
&gt;&gt;&gt; loss.backward()
&gt;&gt;&gt; param_1.grad
tensor([5.])
&gt;&gt;&gt; param_2.grad
tensor([1.3863])

</code></pre>

AbraMunna: <p>Thank you very much, I will have a look at it</p>
Reducing the memory footprint for multiple loss functions

Kimonili: <p>Hi all,</p>
<p>Say I calculate 4 losses for each batch as shown below:</p>
<pre><code class="lang-auto">losses_list = []
some_losses_fns = [loss_fn1,loss_fn2,loss_fn3,loss_fn4]
for loss_fn in some_losses_fns:
  # assume loss is computed 
  loss = loss_fn(preds, true_labels)
  losses_list.append(loss.item())
average_loss = torch.cat(losses_list).mean()
average_loss.requires_grad_()
</code></pre>
<p>Are the gradients of the losses going to be backpropagated properly if when I call <code>.item()</code> on them and then call <code>requires_grad_()</code> on the average loss?</p>
<p>I am trying to reduce the memory footprint of my model and I found this suggestion online but I wanted to double check whether it is legit. Any other suggestions? Maybe <code>del</code> the <code>loss</code> variable after it is appended to the <code>losses_list</code>?</p>
<p>Thanks in advance!</p>

Azerus: <p>Hello,</p>
<p>In short, it won’t work using <code>item</code>. You need to keep tensors all along, else you’ll lose the computational graph (DAG) which is used by the autograd when calling backward (see autograd mechanism in the doc). <code>item()</code> method return a standard python number, which is indeed not compatible with backward.</p>
<p>You can safely use <code>del</code> on the <code>loss</code> after it is appended to the <code>losses_list</code>, but my guess is you won’t notice any significant change in memory footprint.</p>
<p>I don’t see any way to reduce the memory footprint on this part of your code.</p>
<p>Regards,<br>
Thomas</p>
<p>[Edit] By the way, look to the documentation for the <a href="https://pytorch.org/docs/stable/generated/torch.Tensor.item.html" rel="noopener nofollow ugc">item method</a>, it clearly states “This operation is not differentiable.”</p>

Kimonili: <p>Hi <a class="mention" href="/u/azerus">@Azerus</a>,</p>
<p>Thanks for your answer. Yeap, my guess was that the <code>item()</code> method is going to destroy the DAG. I got this from this article <a href="https://blog.paperspace.com/pytorch-memory-multi-gpu-debugging/#:~:text=Using%20Python%20Data%20Types%20Instead%20Of%201%2DD%20Tensors" class="inline-onebox" rel="noopener nofollow ugc">Memory Management, Optimisation and Debugging with PyTorch</a></p>
<p>I found my way around this problem by utilising GPU nodes with more memory, so that my pods are not evicted, but, of course for a higher price <img src="https://discuss.pytorch.org/images/emoji/apple/grin.png?v=10" title=":grin:" class="emoji" alt=":grin:"></p>