akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Where is java API reference for 1.10?

ryos: <p>I cannot find java API reference for 1.10.</p>
<p>I can find API reference only for 1.9.0 ( <a href="https://pytorch.org/javadoc/1.9.0/" class="inline-onebox" rel="noopener nofollow ugc">pytorch_android 1.9.0 API</a> )<br>
which is linked from <a href="https://pytorch.org/mobile/android/#building-pytorch-android-from-source" class="inline-onebox" rel="noopener nofollow ugc">Android | PyTorch</a> .</p>
<p>Where is java API reference for 1.10?</p>

Linbin: <p>Java API was introduced in 1.4</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://pytorch.org/blog/pytorch-1-dot-4-released-and-domain-libraries-updated/#java-bindings-experimental">
  <header class="source">
      <img src="https://pytorch.org/favicon.ico?" class="site-icon" width="32" height="32">

      <a href="https://pytorch.org/blog/pytorch-1-dot-4-released-and-domain-libraries-updated/#java-bindings-experimental" target="_blank" rel="noopener nofollow ugc">pytorch.org</a>
  </header>

  <article class="onebox-body">
    <img src="https://pytorch.org/assets/images/pytorch-logo.png" class="thumbnail onebox-avatar" width="500" height="500">

<h3><a href="https://pytorch.org/blog/pytorch-1-dot-4-released-and-domain-libraries-updated/#java-bindings-experimental" target="_blank" rel="noopener nofollow ugc">PyTorch</a></h3>

  <p>An open source machine learning framework that accelerates the path from research prototyping to production deployment.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
How can I know the size of data_loader when i use: torchvision.datasets.ImageFolder

isalirezag: <p>Im following the example <a href="http://pytorch.org/docs/master/torchvision/datasets.html" rel="nofollow noopener">here</a>, regarding torchvision.datasets.<br>
In the example we have:</p>
<pre><code>imagenet_data = torchvision.datasets.ImageFolder('path/to/imagenet_root/')
data_loader = torch.utils.data.DataLoader(imagenet_data,
                                          batch_size=,
                                          shuffle=True,
                                          num_workers=args.nThreads)
</code></pre>
<p>I am wondering if there is a way that I can know the number of images in data_loader? I want to check its size and see how many images is in it…</p>
<p>Thank you</p>

smth: <p>you can do:</p>
<pre><code class="lang-auto">print(len(imagenet_data))
</code></pre>

Peter_Cha: <p>You can get the length of dataloder’s dataset like this:</p>
<pre><code class="lang-auto">print(len(dataloader.dataset))
</code></pre>

vinaykumar2491: <p>How would we do the same when we use <code>sampler=torch.utils.data.SubsetRandomSampler()</code> when creating the dataloader?</p>
<pre><code class="lang-auto">indices = np.arange(0, len(dataset))
train_dl = DataLoader(dataset, bs, sampler=torch.utils.data.SubsetRandomSampler(indices[:300]))
test_dl  = DataLoader(dataset, bs, sampler=torch.utils.data.SubsetRandomSampler(indices[-300:]))
</code></pre>
<pre><code class="lang-auto">len(train_dl.dataset) == len(test_dl.dataset)
</code></pre>

stroncea: <p><code>len(loader.dataset.indices)</code> should give you the number of images in the dataloader</p>

edgarinvillegas: <p>You can do with:<br>
<code>len(train_loader.sampler.indices)</code></p>

scchess: <p>print(len(dataloader.dataset)) worked</p>
Shape for multiple-channel, 2D mask weights using BCEWithLogitsLoss

morepenguins: <p>I have a set of 256x256 images that are each labeled with nine, binary 256x256 masks. I am trying to calculate the <code>pos_weight</code> in order to weight the <code>BCEWithLogitsLoss</code>.</p>
<p>The shape of my masks tensor is:</p>
<pre><code class="lang-auto">tensor([1000, 9, 256, 256])
</code></pre>
<p>Where 1000 is the number of training images, 9 is the number of mask channels (all encoded to 0/1), and 256 is the size of each image side.</p>
<p>To calculate <code>pos_weight</code>, I have summed the zeros in each mask, and divided that number by the sum of all of the ones in each mask (following the advice suggested <a href="https://discuss.pytorch.org/t/weights-in-bcewithlogitsloss/27452/21">here</a>.):</p>
<pre><code class="lang-auto">(masks[:,channel,:,:]==0).sum()/masks[:,channel,:,:].sum()
</code></pre>
<p>Calculating the weight for every mask channel provides a tensor with the shape of <code>tensor([9])</code>, which seems intuitive to me, since I want a pos_weight value for each of the nine mask channels. However when I try to fit my model, I get the following error message:</p>
<pre><code class="lang-auto">RuntimeError: The size of tensor a (9) must match the size of tensor b (256) at non-singleton dimension 3
</code></pre>
<p>This surprises me because the error message seems to suggest that the weights need to be the size of the side of the images, but not the number of mask channels. What shape should <code>pos_weight</code> be and how do I specify that it should be providing weights for the mask channels?</p>
Faster RCNN Hierarchical Classification Head

JoshWilson: <p>Hello,<br>
I’m using faster-rcnn to detect animals in images and I would like my classification head to output scores for a number of sub-categories, for example it might output 0.9 for animal, then 0.6 for dog, then 0.3 for poodle. I am wondering if it is possible to replace the classification head of Faster RCNN with a hierarchical classification network, such as that described here: <a href="https://github.com/Ugenteraan/Deep_Hierarchical_Classification" class="inline-onebox" rel="noopener nofollow ugc">GitHub - Ugenteraan/Deep_Hierarchical_Classification: PyTorch Implementation of Deep Hierarchical Classification for Category Prediction in E-commerce System</a>. If it is possible, could I get some guidance on how this might be done.</p>
<p>Thank you for your time!</p>
Why limit batchnorm to 3d?

NilsMi: <p>In the <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/cudnn/BatchNorm.cpp" rel="noopener nofollow ugc">source code</a> for batch norm (lines 110 and 283), it checks for dimensions between 2 and 5, but this is already checked in the class definitions on the python source code. This isn’t a major issue, but then you can’t create an N-dimensional batch norm which would be useful in some cases. And the check is already mandatory in the Python source code. If there isn’t a reason why (does it break?), this check could easily be removed and just let python check the dimensions (as is already does), to allow implementation of 4D (or nD) batchnorm easily for example:</p>
<pre><code class="lang-auto">class BatchNorm4d(torch.nn.modules.batchnorm._BatchNorm):
    def __init__(self, num_features, **kwargs):
        super().__init__(num_features=num_features, **kwargs)
    def _check_input_dim(self, input):
        if input.dim() != 6:
            raise ValueError("expected 6D input (got {}D input)".format(input.dim()))
</code></pre>
<p>This is how 1d-&gt;3d batchnorm is already implemented in the python section.</p>
Mask rows in a matrix

JoshP: <p>Hi,</p>
<p>I have a mask vector of binary values, I would like to use this to essentially mask rows in a matrix:</p>
<pre><code class="lang-auto">mask = [1, 0, 1]
matrix = [[1, 2, 3],
          [4, 5, 6],
          [7, 8, 9]]

out = mask * matrix
out =  [[1, 2, 3],
        [0, 0, 0],
        [7, 8, 9]]
</code></pre>
<p>I am trying to figure out how I can implement the <code>out = mask * matrix</code> operation when I have a batch of different masks and 1 matrix, so shapes are:</p>
<pre><code class="lang-auto">mask.shape  = [batch, n]
matrix.shape = [n, m]
out.shape = [batch, n, m]
</code></pre>
<p>I think I might have to use torch.repeat to repeat the matrix over the batch dim but not sure.</p>

JoshP: <p>I actually figured this out straight away. All you need to do is unsqueeze the mask vector and repeat the matrix over the batch dimension:</p>
<pre><code class="lang-auto">batch_size = 2
a = torch.randn(batch_size, 3)
b = torch.randn(3, 4)

a = a.unsqueeze(-1)

b = b.unsqueeze(0)
b = b.repeat(batch_size, 1, 1)

print(a.shape, b.shape)
&gt;&gt;&gt; torch.Size([2, 3, 1]) torch.Size([2, 3, 4])

c = (a * b)
print(c.shape)
&gt;&gt;&gt; torch.Size([2, 3, 4])
</code></pre>
Input argument is an empty tuple, using register_forward_pre_hook

alireza_dizaji: <p>Hi,<br>
I am trying to use register_forward_pre_hook to modify one of my model’s forward input, and it seems the hook cannot recognize any of the input arguments. Here is my code which is a wrapper for the models of a library using keyword arguments for its forward function and I tried to separate two of the arguments which are necessary for modification.</p>
<pre><code class="lang-auto">class GNNWrapper(nn.Module):
    """ the only point of this class is to pass `x` and `edge_index` as separate input
    arguments and therefore torch hooks could recognize them"""
    
    def __init__(self, model: GNNBasic) -&gt; None:
        super().__init__()

        self._model: GNNBasic = model

    def forward(self, x, edge_index, *args, **kwargs):
        kwargs['x'] = x
        kwargs['edge_index'] = edge_index

        return self._model(*args, **kwargs)
</code></pre>
Theory on back propagation: whether the loss function can be defined freely?

GoodManWEN: <p>Hi all, due to lack of basic theory, one thing I am not sure about is, whether the network can back propagate or not depends only on the model design itself, and has nothing to do with the choice of loss function?</p>
<p>This means that as long as the network is properly designed and the loss function can provide a result, it will always optimize the result towards it. So there is only the question of whether the loss function is suitable for the task, without the worry that the network will not be trained due to the wrong choice of loss function?</p>

ptrblck: <aside class="quote no-group" data-username="GoodManWEN" data-post="1" data-topic="157633">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/goodmanwen/48/44146_2.png" class="avatar"> GoodManWEN:</div>
<blockquote>
<p>whether the network can back propagate or not depends only on the model design itself, and has nothing to do with the choice of loss function?</p>
</blockquote>
</aside>
<p>No, the loss function has to be written using differentiable operations or at least be able to yield a valid gradient (the same applies to the model of course as well).</p>
Iterable Dataset in C++

lkc1: <p>Is there an example of writing custom dataset class for <code>IteratrableDataset</code> in C++? Is it even possible? I see <a href="https://github.com/pytorch/examples/blob/main/cpp/custom-dataset/custom-dataset.cpp" rel="noopener nofollow ugc">this example</a> for map-styled dataset.</p>
<p>My goal is to write a custom <code>DataLoader</code> in C++ where multiple workers are working in parallel to fetch samples from a database. If <code>num_workers &gt; 1</code>, does <code>prefetch</code> in C++ work in the same way as in python, except it uses multi-threading and is not subject to the problems of GIL?</p>
Docs for LSTM and GRU

itamblyn: <p>I think the docs for LSTM / GRU need to be modified:</p>
<p><a href="https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html</a></p>
<p>I have noticed that students consistently misunderstand the definition of <code>input_size</code> - they think it means the length of the series. This confuses them, as they expect LSTM to work with variable length data.</p>
<p>Additionally, the “example” at the bottom of the page is not easy to follow.</p>
<p>I’ve seen over 100 people go through these docs at this point, and I would estimate less than 10% of them understand how to use the function after reading them.</p>

ptrblck: <p>Do you think a more detailed explanation of <code>input_size</code> would be beneficial or a change in the argument name (if this would be backwards compatibility breaking, it might not be easy to land)?<br>
In any case, would you be interested in improving the docs as you seem to have a good idea which improvements could be made?</p>
Training accuracy LSTM NN

Uma_Sushmitha_Guntur: <pre><code class="lang-auto">for epoch in range(n_epochs):
        for i in range(1):
            sp_idx = np.random.randint(0, Xtrain.shape[0], batch_size)
            images = Xtrain[sp_idx, :,:]
            labels = ytrain[sp_idx]
            images = images.reshape(-1, time_step, input_size) 
            
            # forward pass
            outputs, _= model(images) 
            loss = criterion(outputs, labels)
            
            # backward and optimize
            optimizer.zero_grad()
            loss.backward()
            torch.nn.utils.clip_grad_norm_(model.parameters(), 5)
            optimizer.step()
            train_losses.append(loss.item())
            print("Train Loss", train_losses)
            outputs, outputs_prob = model(Xtest) #TLNN
            test_loss = criterion(outputs, ytest)
            valid_losses.append(test_loss.item())
            print("Validation Losses", valid_losses)
        
   
            _, predicted = torch.max(outputs_prob.data, 1)
            predicted_prob = outputs_prob[:,1]
            accu = accuracy_score(ytest, predicted)
            cof_mat = confusion_matrix(predicted, ytest)
            
            try:
                accu_all.append(accu)
                auc_all.append(roc_auc_score(ytest.detach().numpy(), predicted_prob.detach().numpy()))
                f1_all.append(f1_score(ytest.detach().numpy(), predicted))
                recall_all.append(recall_score(ytest, predicted))
                precise_all.append(precision_score(ytest, predicted))
            except Exception as e:
                print(e, 'excepted')
                continue
                
        
        train_loss = np.average(train_losses)
        valid_loss = np.average(valid_losses)
        avg_train_losses.append(train_loss)
        avg_valid_losses.append(valid_loss)
        epoch_len = len(str(n_epochs))
</code></pre>
<p>How to find Training accuracy in this neural network? I am trying to plot Training accuracy , training loss ad valid loss for this network. <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

ptrblck: <p>I don’t know what your exact use case is, but you should be able to calculate the accuracy by computing it between the model predictions and the targets.<br>
E.g. for a multi-class classification take a look at the <a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html">CIFAR10 tutorial</a> which calculates the accuracy of the test datset as:</p>
<pre><code class="lang-python">correct = 0
total = 0
# since we're not training, we don't need to calculate the gradients for our outputs
with torch.no_grad():
    for data in testloader:
        images, labels = data
        # calculate outputs by running images through the network
        outputs = net(images)
        # the class with the highest energy is what we choose as prediction
        _, predicted = torch.max(outputs.data, 1)
        total += labels.size(0)
        correct += (predicted == labels).sum().item()

print(f'Accuracy of the network on the 10000 test images: {100 * correct // total} %')
</code></pre>
Training with symmetric matrix

smonsays: <p>I’m trying to train a model that uses a symmetric matrix for the linear layer and wonder how to efficiently implement the symmetric matrix in pytorch.</p>
<p>I have seen <a href="https://discuss.pytorch.org/t/retain-graph-and-training-with-symmetric-matricies/15386">this</a> approach, but I think it does not fulfill my needs since it introduces more trainable parameters than necessary, namely <code>features * features</code> parameters instead of <code>(features * (features+1) ) / 2</code>.</p>
<p>I came up with the following solution:</p>
<pre><code class="lang-auto">features = 3
num_weights = features * (features+1) // 2
weights = torch.randn(num_weights)

tri_mat = torch.zeros(features,features)
tri_idx = torch.tril_indices(features,features)
tri_mat[tri_idx[0], tri_idx[1]] = weights

symmetric_weight = torch.tril(tri_mat) + torch.tril(tri_mat, -1).t()
</code></pre>
<p>However, I suspect creating the temporary <code>tri_mat</code> could be avoided. Any ideas on how to do this more elegantly/efficiently?</p>

rpfeynman: <p>I would enforce it as:</p>
<pre><code class="lang-python">latent_weights = torch.FloatTensor([...])
weights = (latent_weights + torch.transpose(latent_weights))/2
</code></pre>

smonsays: <p>This also introduces <code>features * features</code> tunable parameters instead of <code>(features * (features+1) ) / 2</code> tunable parameters, right? That is what I am trying to avoid here.</p>

arnavs: <p><a class="mention" href="/u/smonsays">@smonsays</a> I’m running into exactly this problem now. Did you ever figure out how to do it without creating the temporary?</p>

jasperhyp: <p>It’s probably too late but one simple way is <code>W.triu() + W.triu(1).transpose(-1, -2)</code>.  This way you only use half of the parameters.</p>
Loss value remains constant during training

ZahAbdel: <p>I’m using a Unet architecture. During training the loss value is the same.<br>
Dataset: 4000 sample from 160 augmented images<br>
I changed the learning rate many time from 0.1 to 0.000001 but the result is the same.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/7/e73c04758c860bdbc6170d1d4d076bbc299804a5.png" data-download-href="https://discuss.pytorch.org/uploads/default/e73c04758c860bdbc6170d1d4d076bbc299804a5" title="2"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/7/e73c04758c860bdbc6170d1d4d076bbc299804a5_2_690x308.png" alt="2" data-base62-sha1="wZASHWFhwT3x8l6wVzf23sZGXRz" width="690" height="308" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/7/e73c04758c860bdbc6170d1d4d076bbc299804a5_2_690x308.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/7/e73c04758c860bdbc6170d1d4d076bbc299804a5_2_1035x462.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/7/e73c04758c860bdbc6170d1d4d076bbc299804a5.png 2x" data-dominant-color="424242"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">2</span><span class="informations">1146×512 132 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Try to overfit a small dataset (e.g. just 10 samples) by playing around with some hyperparameters.<br>
Once this is done try to scale up the use case again.</p>

ZahAbdel: <p>Hi thanks for your suggestion. I tried with a couple of combination of parameters (optimizer, learning rate, patch size, dropout, and batch normalization) but still get the same results. the problem was solved due to the normalization of data images</p>

knitemblazor: <p>So did you go through the entire dataset to pick up the values for normalization? i guess the normalization you are talking about is done using torchvision transforms.</p>

ZahAbdel: <p>I just normalize all dataset. I have a 8 bit images, so<br>
newData = oldData / 255</p>

knitemblazor: <p>ok thanks for the answer</p>
What books have you read to learn ML?

knoriy: <p>Hi,</p>
<p>My university is in the process of purchasing some ML books. I wanted to ask what ML book you found the most helpful and would highly recommend reading.</p>
<p>So far, I’ve enjoyed reading:</p>
<ul>
<li>Deep Learning by Ian Goodfellow - ISBN-10 - 0262035618</li>
<li>Artificial Intelligence Basics: A Non-Technical Introduction by Tom Taulli - ISBN-10 - 1484250273</li>
<li>Python: Beginner’s Guide to Artificial Intelligence: Build applications to intelligently interact with the world around you using Python by Denis Rothman - ISBN-10 - 178995732X</li>
</ul>
<p>Do you recommend any PyTorch-specific books?</p>

ptrblck: <p>You could check <code>Deep Learning with PyTorch</code> which was written by <a class="mention" href="/u/lantiga">@lantiga</a>, <a class="mention" href="/u/elistevens">@elistevens</a>, and <a class="mention" href="/u/tom">@tom</a>, and which I’ve enjoyed reading.</p>
Semantic segmentation with large images and U-Net

Sylvain_Ard: <p>Hello,<br>
someone nice could please adapt this code : <a href="https://www.dropbox.com/s/oskaen5zbwry1nb/train.py?dl=0" class="inline-onebox" rel="noopener nofollow ugc">Dropbox - train.py - Simplify your life</a> to exploit 3500x2500 pixels images at input of neural network :<br>
I have a 24GB RAM GPU, if 3500x2500 pixels is too much large to fit the RAM, please take a smaller dimension.<br>
Thank you<br>
Best regards</p>

Sylvain_Ard: <p>other files needed :</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://www.dropbox.com/s/15h2jeua2yl8po6/snapshot.py?dl=0">
  <header class="source">
      <img src="https://cfl.dropboxstatic.com/static/metaserver/static/images/favicon-vfl8lUR9B.ico" class="site-icon" width="32" height="32">

      <a href="https://www.dropbox.com/s/15h2jeua2yl8po6/snapshot.py?dl=0" target="_blank" rel="noopener nofollow ugc">Dropbox</a>
  </header>

  <article class="onebox-body">
    <img src="https://www.dropbox.com/static/metaserver/static/images/spectrum-icons/generated/content/content-code-large.png" class="thumbnail onebox-avatar" width="160" height="160">

<h3><a href="https://www.dropbox.com/s/15h2jeua2yl8po6/snapshot.py?dl=0" target="_blank" rel="noopener nofollow ugc">snapshot.py</a></h3>

  <p>Shared with Dropbox</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://www.dropbox.com/s/cvkk0ak706fj9bx/seg_loss.py?dl=0">
  <header class="source">
      <img src="https://cfl.dropboxstatic.com/static/metaserver/static/images/favicon-vfl8lUR9B.ico" class="site-icon" width="32" height="32">

      <a href="https://www.dropbox.com/s/cvkk0ak706fj9bx/seg_loss.py?dl=0" target="_blank" rel="noopener nofollow ugc">Dropbox</a>
  </header>

  <article class="onebox-body">
    <img src="https://www.dropbox.com/static/metaserver/static/images/spectrum-icons/generated/content/content-code-large.png" class="thumbnail onebox-avatar" width="160" height="160">

<h3><a href="https://www.dropbox.com/s/cvkk0ak706fj9bx/seg_loss.py?dl=0" target="_blank" rel="noopener nofollow ugc">seg_loss.py</a></h3>

  <p>Shared with Dropbox</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


Sylvain_Ard: <p>noone could do that please ?</p>

ptrblck: <p>As with your previous posts I don’t think you are giving  enough information about your issue, where you are stuck, and especially what you’ve tried so far to solve it.<br>
I don’t think just asking users to implement your request is too compelling.</p>
How to use ignore_index in torch.nn.CrossEntropyLoss

mmmartin: <p>Given a sequence: <code>a b c [SEP] d e f</code>,<br>
the corresponding labels is : <code>-1 -1 -1 0 1 0 0</code>.</p>
<p>I want to compute the loss <code>loss_func(logits,labels,ignore_index=-1)</code> to ignore the tokens before <code>[SEP]</code>.<br>
And I got a <code>RuntimeError: multi-target not supported at /opt/conda/conda-bld/pytorch_1603729096996</code>.</p>
<p>So I have to change labels to “class indice mode” like <code>4</code>, but how to ignore the previous tokens when calculating loss.</p>

ptrblck: <p>Could you post a minimal, executable code snippet using random tensors in your shapes, which would reproduce the issue?<br>
Based on the error message I would guess you are using <code>nn.CrossEntropyLoss</code> and are passing the <code>target</code> in the wrong shape.</p>
Sigmoid vs Binary Cross Entropy Loss

Aldebaran: <p>In my torch model, the last layer is a <code>torch.nn.Sigmoid()</code> and the loss is the <code>torch.nn.BCELoss</code>.<br>
In the training step, the following error has occurred:</p>
<pre><code class="lang-python">RuntimeError: torch.nn.functional.binary_cross_entropy and torch.nn.BCELoss are unsafe to autocast.
Many models use a sigmoid layer right before the binary cross entropy layer.
In this case, combine the two layers using torch.nn.functional.binary_cross_entropy_with_logits
or torch.nn.BCEWithLogitsLoss.  binary_cross_entropy_with_logits and BCEWithLogits are
safe to autocast.

</code></pre>
<p>However, when trying to reproduce this error while computing the loss and backpropagation, everything goes correctly:</p>
<pre><code class="lang-python">import torch
from torch import nn

# last layer
sigmoid = nn.Sigmoid()
# loss
bce_loss = nn.BCELoss()


# the true classes
true_cls = torch.tensor([
            [0.],
            [1.]])

# model prediction classes
pred_cls = sigmoid(
    torch.tensor([
           [0.4949],
           [0.4824]],requires_grad=True)
)
pred_cls
# tensor([[0.6213],
#         [0.6183]], grad_fn=&lt;SigmoidBackward&gt;)

out = bce_loss(pred_cls, true_cls)
out
# tensor(0.7258, grad_fn=&lt;BinaryCrossEntropyBackward&gt;)

out.backward()
</code></pre>
<p>What am i missing?<br>
I appreciate any help you can provide.</p>

ptrblck: <p>In the posted code snippet you are not using mixed-precision training and in particular <code>torch.cuda.amp.autocast</code>, which raises the initial error.<br>
The reason for this error is the numerical stability of <code>sigmoid</code> + <code>nn.BCELoss</code>, which is already less stable than <code>nn.BCEWithLogitsLoss</code> in <code>float32</code> and could suffer more when using mixed-precision training.</p>

zestyzapus: <p>The posted code doesn’t raise an error anymore in Pytorch, but it still raise the same error when I use Pytorch Lightning with mixed precision. So indeed we should always use <code>nn.BCEWithLogitsLoss</code> other than <code>sigmoid</code> + <code>nn.BCELoss</code>.</p>

ptrblck: <p>I don’t think the posted code was using <code>amp</code>, but this code would still raise the error:</p>
<pre><code class="lang-python">import torch
import torch.nn as nn


x = torch.sigmoid(torch.randn(10, 1, requires_grad=True, device='cuda'))
y = torch.randint(0, 2, (10, 1), device='cuda').float()
criterion = nn.BCELoss()

with torch.cuda.amp.autocast():
    loss = criterion(x, y)

loss.backward()
</code></pre>
<p>In any case, you are right that <code>nn.BCEWithLogitsLoss</code> should be used for numerical stability reasons.</p>
Custom function in Pytorch using outside libraries

s_mkb: <p>Hello. I want to create a neural network that has a loss function coming from two sources: the output of the neural network, as well as a function applied to this output. Basically, this function takes in values that are coming from the neural network, and produces a set of new values. I want to simultaneously take into account losses coming directly from the output of the neural network AND the output of the function given this prediction, to generate a prediction that gives me the best function output.</p>
<p>For instance:</p>
<p>Loss = MSE(prediction, target) + MSE(function(pred), function(target))</p>
<p>The important thing to note is that this function is coming from an external python library, and it is not something I can recreate in Pytorch. Would this work? What can I do so that it works as a Pytorch function?</p>

ptrblck: <p>You can create a custom <code>autograd.Function</code> and implement the <code>forward</code> as well as the <code>backward</code> for this operation using the 3rd party library as seen <a href="https://pytorch.org/tutorials/beginner/examples_autograd/polynomial_custom_function.html">here</a>.</p>

s_mkb: <p>Thank you! How about if the function was not known? Basically, I’m going to pass some input to a function from an external python library, and it will give me an output. I’m not entirely sure what I would put in the backwards method. Would it just be the gradient of the loss with respect to the output?</p>

ptrblck: <p>In that case I wouldn’t know if using this method in the forward pass would make sense.<br>
Your suggestion of returning a static gradient might technically work, but since it wouldn’t reflect the true gradient I don’t know what your model would learn from it.</p>
RuntimeError: torch.triangular_solve: Expected b to have at least 2 dimensions, but it has 1 dimensions instead

ZSheikhb: <p>I converted this function from tensorflow to pytorch</p>
<pre><code class="lang-auto">import tensorflow as tf
def gaussian_log_density(samples, means, chol_covars):
    covar_logdet = 2 * tf.reduce_sum(tf.math.log(tf.linalg.diag_part(chol_covars)), -1)
    diff = tf.expand_dims(samples - means, -1)
    exp_term = tf.reduce_sum(tf.square(tf.linalg.triangular_solve(chol_covars, diff)), axis=[-2, -1])
    return - 0.5 * (tf.cast(tf.shape(samples)[-1], exp_term.dtype) * np.log(2 * np.pi) + covar_logdet + exp_term)
</code></pre>
<p>and I am wondering whether I correctly converted <code>tf.linalg.diag_part</code> in <code>tf.reduce_sum(..., axis=[-2, -1])</code> in my implementation?</p>
<pre><code class="lang-auto">def gaussian_log_density(samples, means, chol_covars):
    covar_logdet = 2 *torch.sum(torch.log(torch.diag_embed(chol_covars, offset=0, dim1=-2, dim2=-1)+1e-10),dim=-1)
    diff = torch.unsqueeze(samples - means, -1)
    exp_term = torch.sum(torch.square(torch.triangular_solve(chol_covars, diff)), (-2, -1))
    return - 0.5 * (samples.size()[-1].type( exp_term.dtype) * torch.log(2 * torch.tensor(np.pi)) + covar_logdet + exp_term)

if __name__=="__main__":
  a = torch.tensor([2.0, 1.1, 1.2])
  b =torch.tensor([1, 2.4, 5.35])
  s= torch.tensor([1.02, 0.9, 1.1])
  print(gaussian_log_density(s, a, b))
</code></pre>
<p>since I am getting this error just with testing my code:</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-44-ab80d557c449&gt; in &lt;module&gt;()
     10   b =torch.tensor([1, 2.4, 5.35])
     11   s= torch.tensor([1.02, 0.9, 1.1])
---&gt; 12   print(gaussian_log_density(s, a, b))

&lt;ipython-input-44-ab80d557c449&gt; in gaussian_log_density(samples, means, chol_covars)
      3     print(covar_logdet)
      4     diff = torch.unsqueeze(samples - means, -1)
----&gt; 5     exp_term = torch.sum(torch.square(torch.triangular_solve(chol_covars, diff)), (-2, -1))
      6     return - 0.5 * (samples.size()[-1].type( exp_term.dtype) * torch.log(2 * torch.tensor(np.pi)) + covar_logdet + exp_term)
      7 

RuntimeError: torch.triangular_solve: Expected b to have at least 2 dimensions, but it has 1 dimensions instead
</code></pre>
<p>Thanks.</p>