akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Fill tensor with another tensor where mask is true

StefanoBerti: <p>I need to insert elements of tensor <em>new</em> into a tensor <em>old</em> with a certain probability, let’s say that it is 0.8 for simplicity. Substantially this is what masked_fill would do, but it only works with monodimensional tensor. Actually I am doing</p>
<pre><code class="lang-auto">    prob = torch.rand(trgs.shape, dtype=torch.float32).to(trgs.device)
    mask = prob &lt; 0.8

    dim1, dim2, dim3, dim4 = new.shape
    for a in range(dim1):
        for b in range(dim2):
            for c in range(dim3):
                for d in range(dim4):
                    old[a][b][c][d] = old[a][b][c][d] if mask[a][b][c][d] else new[a][b][c][d]
</code></pre>
<p>which is awful. I would like something like</p>
<pre><code class="lang-auto">    prob = torch.rand(trgs.shape, dtype=torch.float32).to(trgs.device)
    mask = prob &lt; 0.8

    old = trgs.multidimensional_masked_fill(mask, new)
</code></pre>

Eta_C: <pre><code class="lang-python">def old_imp(old, new, mask):
    dim1, dim2, dim3, dim4 = new.shape
    for a in range(dim1):
        for b in range(dim2):
            for c in range(dim3):
                for d in range(dim4):
                    old[a][b][c][d] = old[a][b][c][d] if mask[a][b][c][d] else new[a][b][c][d]
    return old 

def new_imp(old, new, mask):
    new[mask] = old[mask]
    return new

old = torch.rand(2, 3, 4, 5)
new = torch.rand(2, 3, 4, 5)
prob = torch.rand(old.shape, dtype=torch.float32)
mask = prob &lt; 0.8

old_res = old_imp(old.clone(), new.clone(), mask.clone())
new_res = new_imp(old.clone(), new.clone(), mask.clone())
print(old_res ==  new_res)
</code></pre>
Handling large 3d image dataset with DataLoader

Siddharth_Tandon: <p>Hi,</p>
<p>I am working with MSD (Medical Segmentation Decathlon) dataset for a segmentation problem.<br>
In Dataset’s <strong>init</strong> method I am storing all filenames in list, and in <strong>getitem</strong> reading the files, applying transformation, etc…</p>
<p>I am facing some issue with GPU utilization:<br>
Problem is, nvidia-sme command is showing only 1 process (whereas I have specified 4 workers in dataloader) running, with GPU memory usage as 4GB. Secondly there is no GPU utilization at all.</p>
<p>I was reading an article which says that if DataLoader is slow, model will run on CPU. But how can I speed up the dataloader and run everything on GPU</p>

ptrblck: <aside class="quote no-group" data-username="Siddharth_Tandon" data-post="1" data-topic="115184">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/siddharth_tandon/48/32526_2.png" class="avatar"> Siddharth_Tandon:</div>
<blockquote>
<p>I am facing some issue with GPU utilization:<br>
Problem is, nvidia-sme command is showing only 1 process (whereas I have specified 4 workers in dataloader) running, with GPU memory usage as 4GB. Secondly there is no GPU utilization at all.</p>
</blockquote>
</aside>
<p>The <code>num_workers</code> specified in the <code>DataLoader</code> are executed on the CPU using multiprocessing and will not be shown on <code>nvidia-smi</code>.<br>
The GPU utilization might be low, if you are facing bottlenecks in your code, such as data loading.<br>
Take a look at <a href="https://discuss.pytorch.org/t/how-to-prefetch-data-when-processing-with-gpu/548/19">this post</a>, which explains it well and suggests some performance improvements.</p>
<aside class="quote no-group" data-username="Siddharth_Tandon" data-post="1" data-topic="115184">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/siddharth_tandon/48/32526_2.png" class="avatar"> Siddharth_Tandon:</div>
<blockquote>
<p>I was reading an article which says that if DataLoader is slow, model will run on CPU.</p>
</blockquote>
</aside>
<p>This is not correct. The model will use the specified device and PyTorch will not automatically use the CPU based on the performance of the <code>DataLoader</code>.</p>
How to pass the user input as an object to a neural network?

Helen_R6: <p>Hi,<br>
I’ve used hook function to extract the pertained weights at certain layer.</p>
<pre><code class="lang-auto">activation = {}
def get_activation(name):
    def hook(model, input, output):
        activation[name] = output.requires_grad_()
    return hook
</code></pre>
<p>Now I would like to make a user interface that the user will input a layer name (eg. “layer2”, “layer3”), and then pass it as an object to the model.<br>
When the user input ‘layer2’, it will do<br>
<code>resnet_model.layer2.register_forward_hook(get_activation('layer2'))</code><br>
When the user input ‘layer3’, it will do<br>
<code>resnet_model.layer3.register_forward_hook(get_activation('layer3'))</code></p>
<p>My problem is that I don’t know how to pass the string input by user as an object to a nn.</p>
<p>Any help and suggestions would be appreciated, thanks in advance.</p>

ptrblck: <p>You could add this argument to the <code>__init__</code> method or a custom class method of your model:</p>
<pre><code class="lang-python">class MyModel(nn.Module):
    def __init__(self, user_input=''):
        super(MyModel, self).__init__()
        print('user_input is {}'.format(user_input))
        # create the modules here and register the hook
    
    def my_custom_method(self, user_input=''):
        print('user_input is now {}'.format(user_input))
        
    def forward(self, x):
        return x

model = MyModel(user_input='a')
model.my_custom_method(user_input='b')
</code></pre>
How to use amp in GAN

111220: <p>Generally speaking, the steps to use amp should be like this：</p>
<pre><code class="lang-auto">scaler.scale(loss).backward()
scaler.step(optimizer)
scaler.update()
</code></pre>
<p>This is  my GAN code   <em><strong>(before using amp)</strong></em>:</p>
<pre><code class="lang-auto">optimizer1=SGD(model.D.parameters()......)
optimizer2=SGD(model.G.parameters().....(
for I in range(D_training_steps):
     ........
     loss1.backward()
    optimizer1.step()
    ......

     loss2.backward()      ## after update the D above,I will compute loss2 and update again
     optimizer1.step()
    ....
for I in range(G_training_steps):
    loss3.backward()
    optimizer3.step()

</code></pre>
<p>So the question is when to use scaler.update() if using amp？<br>
Thanks for your help!</p>

ptrblck: <p><a href="https://pytorch.org/docs/stable/notes/amp_examples.html#working-with-multiple-models-losses-and-optimizers">This example</a> should work for your use case. You can use a single <code>scaler</code> for both losses or create separate ones, if you want.</p>
Modifying BatchNorm source code

loganfrank: <p>Hi,</p>
<p>I’m wanting to modify the PyTorch C/C++ source code for Batch (and Group, Layer, etc.) Norm layers for part of my research, which could hopefully result in a contribution to PyTorch if successful and the work is substantial.</p>
<p>I’m wondering what files I should look at for modifying? The hope is I can do something like <code>nn.BatchNorm2d(num_features, my_extra_parameter=True)</code> and it would pass my extra parameter down to ultimately where the normalization and (optional) affine transformation is computed in the code. If <code>my_extra_parameter == False</code> then it should behave like the normal batch norm already implemented in PyTorch, if <code>True</code> it should behave differently, following my modifications. I’m wanting to make this work for CPU and GPU (mainly care about GPU since this would be used in training). What I’m trying to do involves already built-in PyTorch functions like abs, sign, etc. which I would think should make making the modifications to the forward and backward functions of batch norm relatively easy. I just don’t know where to look.</p>
<p>Thank you!</p>

ptrblck: <p>You can find the CUDA kernels <a href="https://github.com/pytorch/pytorch/blob/c618dc13d2aa23625cb0d7ada694137532a4fa33/aten/src/ATen/native/cuda/Normalization.cuh">here</a>, but they might not be straightforward to understand.<br>
<a href="https://github.com/pytorch/pytorch/blob/8c798e062216278673a75bac0848ea69a8bd3f03/aten/src/ATen/native/Normalization.cpp">Here</a> are the CPU implementations, in case you want to also take a look at it.</p>
<p>I would personally recommend to write your experiments in a custom (Python) module first and verify the usability in a few experiments. A manual batchnorm implementation can be found <a href="https://github.com/ptrblck/pytorch_misc/blob/master/batch_norm_manual.py">here</a>.</p>

loganfrank: <p>Thank you for the response! I do have it working as a custom Python module currently and have some initial results that show potential. From what we have seen, we are wanting to explore further. My advisor is wanting me to modify the source C++ code so that whatever we would present in a paper is exactly the behavior one would expect if our method was implemented into PyTorch and another person uses it.</p>
<p>I’m curious if I were to rewrite the normal PyTorch batch norm layer using <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" rel="noopener nofollow ugc">the PyTorch CPP extension</a>, should I be able to exactly match the performance of PyTorch’s version? If that’s the case, it seems like it’d be easier to implement it that way, but we would also be able to get close enough to what my advisor is wanting.</p>

ptrblck: <p>Yes, I think using a custom C++/CUDA extension should not introduce a performance penalty compared to built-in methods.</p>
Increasing the values for specific indices in a tensor

mel: <p>I want to increase specific indices in a tensor by 10%.<br>
I have a tensor</p>
<pre><code class="lang-auto">x = torch.randint(-2,2,[4,3])

tensor([[-2,  0,  1],
        [-2,  1, -2],
        [ 0,  0, -1],
        [ 0, -1,  1]])
</code></pre>
<p>and the following indices</p>
<pre><code class="lang-auto">indices = torch.tensor([[0,0,1],[2,1,1]])
</code></pre>
<p>I want to increase the values by 10% on the indices above, so that I get the following tensor</p>
<pre><code class="lang-auto">tensor([[-2,  0,  1.1],
        [-2,  1.1, -2],
        [ 0,  0, -1],
        [ 0, -1,  1]])
</code></pre>
<p>How can I achieve that?</p>

albanD: <p>Hi,</p>
<p>I guess you can do something like this no?<br>
<code>x[indices[0], indices[1]] *= 1.1</code></p>

mel: <p>HI <a class="mention" href="/u/alband">@albanD</a> thank you for the response.<br>
I tried your solution but I lose the full matrix, ang get only the selected vector</p>
<pre><code class="lang-auto">x[indices[0], indices[1]]

&gt;&gt; tensor([ 1, -2, -1])
</code></pre>
<p>Also, multiplying by negative numbers by 1.1 will reduce them and instead of increasing them<br>
so  I guess I need a calculation similar to</p>
<pre><code class="lang-auto">x + x.abs() * 0.1
</code></pre>
<p>I just don’t know how to apply it only on the specific indices.</p>

albanD: <p>Ho in that case you can do:</p>
<pre><code class="lang-auto">vals = x[indices[0], indices[1]]
updated_vals = vals + vals.abs() * 1.1
x[indices[0], indices[1]] = vals
</code></pre>

mel: <p>Thank you <a class="mention" href="/u/alband">@albanD</a>, just what I was looking for</p>
How to retain updated parameters on the actual model when using Higher?

TinfoilHat0: <p>Hi,</p>
<p>I’m using higher library to develop some meta-learning algorithm.<br>
Briefly, my code looks like the following:</p>
<pre><code class="lang-auto">with higher.innerloop_ctx(model, opt) as (fmodel, diffopt):
    for xs, ys in data:
        logits = fmodel(xs)  # modified `params` can also be passed as a kwarg
        loss = loss_function(logits, ys)  # no need to call loss.backwards()
        diffopt.step(loss) 
        // update my hyper-parameters

// update model by using hyperparams learned in the higher context.
</code></pre>
<p>My problem is, I’d like the diffopt to update my actual model as well as fmodel. To do so, I’m currently doing another update step after learning my hyperparams in the higher context, and thus using an extra forward-backward propagation.</p>
<p>Is such a thing possible?</p>
Overlapping Issues

moreshud: <p>I am working on synthetic 3D medical image dataset with mainly 2 classes with spatial dimension 512 * 512 and varying depth i.e <code>y * 512*512</code>. My approach is to train with 2D Unet and generated the background by stacking the one-hot encoded target of the corresponding pixels of each class - making 1’s for the background pixel value where both classes are zero and 0’s where either of the classes is one.</p>
<p>The data is highly imbalanced and there are overlaps with the main classes except for the background considering the way is generated. Having trained severally with unreasonable results - I mean the model not learning, I gathered insights from similar and related discussions on the forum which revealed that I am dealing with multi-class multilabel segmentation. To this, I used <code>BCEWithLogitsLoss</code> passing the logits and stacked one-hot encoded target maintaining dimension of <code>nBatch, nClass, Width, Heights</code> using a batch size of 8 which resulted in a decreasing loss i.e the model is training.</p>
<p>I would be glad for your clarification and explanation regarding:</p>
<ul>
<li>choice of weighted loss function - which is applicable pos_weight or weight of the <code>BCEWithLogitsLoss</code>, can other weighted loss functions be considered</li>
<li>Resizing the spatial dimension for the image and target as I am short of computational resources</li>
<li>Handling the evaluation metrics as currently using the DiceMetric from monai.</li>
<li>other helpful insights and suggestion</li>
</ul>
[Pytorch Geometric] Question over how to pass in data from DataLoader

fherder: <p>Hello, I’m a newbie to playing around with pytorch geometric, and I’ve made the following toy code (it’s based on a real problem I’m working on):</p>
<pre><code class="lang-auto">import pandas as pd
import numpy as np
import argparse
import torch
from torch_geometric.nn import EdgeConv
from torch_geometric.data import Data, DataLoader
from torch.nn import Sequential, Linear

X = torch.randn(1000,4,4)
Y = torch.randn(1000,5000)

graphs = []
for vector in X:
    x = torch.tensor([[i] for i in vector.flatten()])
    edges = []
    node_index = 0
    for i in range(4):
        for j in range(4):
            if (j != 3): edges.append( [node_index, node_index + 1] )
            if (j != 0): edges.append( [node_index, node_index - 1] )
            if (i != 3): edges.append( [node_index, node_index + 4] )
            if (i != 0): edges.append( [node_index, node_index - 4] )
            node_index += 1
    edges = torch.tensor(edges, dtype = torch.long).t().contiguous()
    graphs.append( Data(x = x, edge_index = edges) )
loader = DataLoader(graphs, batch_size = 256)

m1 = Sequential(
    Linear(2,1)
)

m2 = Sequential(
    Linear(2,1)
)

class Net(torch.nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = EdgeConv(m1, aggr = 'mean')
        self.conv2 = EdgeConv(m2, aggr = 'mean')
    
    def forward(self, data):
        x, edge_index = data.x, data.edge_index

        x = self.conv1(x, edge_index)
        x = self.conv2(x, edge_index)
        x = x.t()

        return Linear(16, 4624)(x)

model = Net()
optimizer = torch.optim.Adam(model.parameters(), lr=0.01, weight_decay=5e-4)
loss = torch.nn.MSELoss()

model.train()
for epoch in range(200):
    optimizer.zero_grad()
    i = 0
    for train_x in loader:
        optimizer.zero_grad()
        out = model(train_x)
        i += 1
    print(i)
</code></pre>
<p>When I run this code using torch == 1.4.0 and torch_geometric == 1.6.0, with Python 3.7, I get the following error:</p>
<pre><code class="lang-auto">RuntimeError: size mismatch, m1: [1 x 4096], m2: [16 x 4624]
</code></pre>
<p>Could someone educate me as to what I’m doing wrong? I know that if I run the Net() on an individual graph it works just fine, however it appears the issue is with how PyTorch Geometric is passing data from the batched graphs. I know it does something different when creating minibatches using DataLoader to allow for efficient mini-batch computation, but it’s not clear from the documentation or examples how exactly we’re supposed to work with these batches. I’ve looked high and low to try and find samples showing how to work with PyTorch Geometric’s DataLoader class, but I can’t seem to find any good references.</p>
<p>Apologies if this is a very elementary question, I’m fairly new at PyTorch in general (I’m coming from TensorFlow) and while I see how some of the concepts line up I don’t think I know enough to effectively debug this problem on my own. Any help would be greatly appreciated.</p>

Li_Zheng: <p>Hi, I’m having exactly the same problem here. Did you find a solution to that? Thanks</p>
Generic torchvision.transform.Compose

escorciav: <p>I wonder if there are any disadvantages of rewriting the default <code>Compose</code> as shown below?</p>
<pre><code class="lang-python">class Compose:
    "Similar to torchvision.transforms.Compose but digest additional kwargs"

    def __init__(self, transforms):
        self.transforms = transforms

    def __call__(self, x, **kwargs):
        for t in self.transforms:
            out = t(x, **kwargs)
            if isinstance(out, Tuple):
                x, kwargs = out
            else:
                x = out
        if isinstance(out, Tuple):
            return x, kwargs
        return x

    def __repr__(self):
        format_string = self.__class__.__name__ + '('
        for t in self.transforms:
            format_string += '\n'
            format_string += '    {0}'.format(t)
        format_string += '\n)'
        return format_string
</code></pre>
<p>For something more specialized, review albumination Compose <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> .</p>
Normalizing a tensor along a dimension

sugh7020: <p>I have a tensor X of shape [B, 3 , 240, 320] where B represents the batch size 3 represents the channels, 240 the height, 320 the width.</p>
<p>I need to find the norm along the channels dimension(3channels) and normalize along that dimension, i.e. each subtensor comprising of the 3 channels should have norm 1.</p>
<p>Can you all please suggest how I can do that?</p>
<p>Thanks!</p>

AlphaBetaGamma96: <p>when you say “norm” which norm do you mean? Standard norm? Softmax norm? Euclidean norm?</p>

sugh7020: <p>Thanks for replying <a class="mention" href="/u/alphabetagamma96">@AlphaBetaGamma96</a></p>
<p>L2 norm</p>

AlphaBetaGamma96: <p>In that case, you should be able to try this!</p>
<pre><code class="lang-auto">X = torch.randn(B,3,240,320)
norm = X.pow(2).sum(dim=1).sqrt()
Xnorm = X/norm
</code></pre>
<p>The L2-norm (or Euclidean norm) is just the square root of the sum of the squares, i.e. norm = sqrt(x^2 + y^2 + …). In your case, you just need to squared the Tensor (via .pow(2)) then sum along the dimension you wish to normalize (via .sum(dim=1)), then take the square root (via .sqrt() ). That calculates your normalization constant. Then just divide the original Tensor by that value and it should normalize your Tensor along that dimension!</p>

sugh7020: <p><a class="mention" href="/u/alphabetagamma96">@AlphaBetaGamma96</a><br>
I tried:</p>
<pre><code class="lang-auto">Xnumerator = torch.cross(Xleft, Xright, dim=1)
norm2 = Xnumerator.pow(2).sum(dim=1).sqrt()
nu = Xnumerator / norm2
</code></pre>
<p>I am getting this error:</p>
<blockquote>
<p>nu = Xnumerator / norm2<br>
RuntimeError: The size of tensor a (3) must match the size of tensor b (60) at non-singleton dimension 1</p>
</blockquote>
<p><em>Xnumerator → torch.Size([60, 3, 240, 320])</em><br>
<em>norm2 → torch.Size([60, 240, 320])</em></p>

AlphaBetaGamma96: <p>This is because your Tensors have different sizes,</p>
<p>You need to add the <code>keepdim=True</code> argument within your sum to keep the dimensions the same size. This works,</p>
<pre><code class="lang-auto">Xnumerator = torch.cross(Xleft, Xright, dim=1)
norm2 = Xnumerator.pow(2).sum(keepdim=True, dim=1).sqrt()
nu = Xnumerator / norm2
</code></pre>
Inserting element in sparse COO tensor

SimonTommerup: <p>Suppose I have a 3x3 adjacency matrix A in sparse format such that</p>
<pre><code class="lang-auto">indices = torch.tensor([[1, 0, 1, 2, 0, 2], [0, 1, 2, 1, 2, 0]])
values = torch.tensor([1., 1., 1., 1., 1., 1.])
A = torch.sparse_coo_tensor(indices, values, size=(3, 3))
</code></pre>
<p>In dense format this would look like</p>
<pre><code class="lang-auto">A.to_dense()
tensor([[0., 1., 1.],
        [1., 0., 1.],
        [1., 1., 0.]])
</code></pre>
<p>I want the graph to be dynamic i.e. new edges can be created. Is there any way to add an edge to a sparse matrix without recreating the sparse matrix from an extended indices and values list?</p>
<p>I have tried e.g.</p>
<pre><code class="lang-auto">A[0,0] = 1
</code></pre>
<p>but this only works if first converting to dense and then inserting:</p>
<pre><code class="lang-auto">B = A.to_dense().clone()
B[0,0] = 5
print(B)
tensor([[5., 1., 1.],
        [1., 0., 1.],
        [1., 1., 0.]])
</code></pre>
<p>I figure that the latter option may be computationally heavy, so what is the best way to do this?</p>
<p>Thanks in advance for any help on this matter<br>
Simon</p>
Neural Language Processing with just one time initialization

Aliaska: <p>Hi everyone,</p>
<p>Could anyone help me how I can train a model in Pytorch with a hidden layer for all the sequences that just one time is initialized with zero at the first forward pass and then not be reseted to zero for each sequence? A simple code would be highly appreciated!!</p>

Dwight_Foster: <p>You can have the forward function return the hidden state as well as the output state and then pass it back into the function every time.</p>
<pre><code class="lang-auto">#INITIALIZE HIDDEN STATE OUTSIDE OF LOOP
def forward(self, input, hidden):
  
   #FORWARD PASS HERE
   return output, hidden
</code></pre>
<p>then each time you can just pass the returned hidden back into the forward function.</p>

Aliaska: <aside class="quote no-group" data-username="Dwight_Foster" data-post="2" data-topic="115227">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dwight_foster/48/18182_2.png" class="avatar"> Dwight_Foster:</div>
<blockquote>
<p><code>hidden</code></p>
</blockquote>
</aside>
<p>Thanks, so this way the hidden would be treated as an input</p>

Dwight_Foster: <p>You should do it outside of your epoch loop like this:</p>
<pre><code class="lang-auto">hidden = #your hidden initialization function or variable
for e in range(epochs)
</code></pre>

Aliaska: <p>Super clear, thanks, I hope it will be resolved</p>
PyTorch AMP matmul inf

xian_kgx: <p>Hi, I’m trying to use PyTorch AMP after knowing that it is now out. However, I’m getting NaN values in my model output. I traced this problem to the output of matmul having some infinite values. The inputs to the matmul operation are fine.</p>
<p>More particularly, I’m using the non-local block from here: <a href="https://github.com/AlexHex7/Non-local_pytorch/blob/master/Non-Local_pytorch_0.4.1_to_1.1.0/lib/non_local_embedded_gaussian.py#L76" class="inline-onebox" rel="noopener nofollow ugc">Non-local_pytorch/non_local_embedded_gaussian.py at master · AlexHex7/Non-local_pytorch · GitHub</a></p>
<p>How do I deal with this?</p>

crcrpar: <p>Hi,</p>
<p>Does the same error happen in normal FP32 training?<br>
If not, I think running the block in <code>autocast(enabled=False)</code> context is one choice. But use this context, it’ll be necessary to convert some input tensors to FP32.</p>
<blockquote>
<p><code>autocast(enabled=False)</code> subregions can be nested in autocast-enabled regions. Locally disabling autocast can be useful, for example, if you want to force a subregion to run in a particular <code>dtype</code>. Disabling autocast gives you explicit control over the execution type. In the subregion, inputs from the surrounding region should be cast to <code>dtype</code> before use: (quote from <a href="https://pytorch.org/docs/stable/amp.html" class="inline-onebox" rel="noopener nofollow ugc">Automatic Mixed Precision package - torch.amp — PyTorch 2.1 documentation</a>)</p>
</blockquote>

ali_safaya: <p>Hi,</p>
<p>A possible solution is to scale-down the values one of the two matrices, or both of them before doing <code>matmul</code> operation, you can try something similar to what in Softmax-Attention.</p>
<p>for example you can try doing it this way:</p>
<pre><code class="lang-auto">f = torch.matmul(theta_x / math.sqrt(your_hidden_size), phi_x)
</code></pre>

xian_kgx: <p>Ya, I shall try that and report back if I found anything. Thanks.</p>
Integrated Gradients for RNNs

christios: <p>Hello,</p>
<p>I’m trying to implement Integrated Gradients (explainability method) for my seq2seq NMT model since there are no public implementations available. For that, I would be required to compute the gradient of my output w.r.t. my input, which would in turn require me to set <code>requires_grad = True</code> for my input variables. Consequently, I would need my input to be of <code>dtype = torch.float</code> (or else an error is thrown), but then I would not be able to use the inputs as inputs to the encoder/decoder embedding layers since those require a <code>dtype = torch.long</code> input. Any suggestions on how to go about this?</p>
<p>Thanks</p>

albanD: <p>Hi,</p>
<p>You won’t be able to get gradients wrt to the input of the embedding layer I’m afraid. Since, as you pointed out, they are not of contiguous dtype.<br>
You might want to do use that technique on the output of the embedding layer instead?</p>

christios: <p>Thanks for your reply, you are right!</p>
Differentiable AUC-proxy Loss Functions

FuriouslyCurious: <p>Several papers have demonstrated that minimizing cross entropy or MSE does not necessarily maximize the area under the ROC curve (AUC).  Are there any differentiable loss functions in PyTorch that can be used as a proxy for AUC?</p>
<p>Two papers have excellent proposals:</p>
<ol>
<li>ICML 2003 - Approximation to the Wilcoxon-Mann-Whitney Statistic  <a href="https://www.semanticscholar.org/paper/Optimizing-Classifier-Performance-via-an-to-the-Yan-Dodier/df27dde10589455d290eeee6d0ae6ceeb83d0c6b?p2df" rel="noopener nofollow ugc">Paper link here</a>
</li>
<li>Scalable Learning of Non-Decomposable Objectives: A new Arxiv 2020 paper by Elad @ Google managed to bump AUC-PR from 84% to 95%.  <a href="https://arxiv.org/abs/1608.04802" rel="noopener nofollow ugc">Paper link here</a>
</li>
</ol>
<p>(1) is <a href="https://github.com/keras-team/keras/issues/1732" rel="noopener nofollow ugc">available as a loss function on Keras</a>, and I am wondering if something similar is available in the PyTorch ecosystem?</p>
<p>Thanks!</p>

phucdoitoan: <p>This would be interesting! Any progress yet?!</p>

Neeks: <p>Something along this line is here … <a href="https://github.com/iridiumblue/roc-star" class="inline-onebox" rel="noopener nofollow ugc">GitHub - iridiumblue/roc-star: Loss function which directly targets ROC-AUC</a></p>
Multiple loss functions

Hdk: <p>Hello,</p>
<p>I am working on a problem where I am using two loss functions together i.e.<br>
Total_loss = cross_entropy_loss + custom_ loss<br>
And then Total_ loss.backward().</p>
<p>The value of Cross entropy loss for a training of say 20 epochs, reaches to ~0.7 from 2.4. It converges faster till approx. 8th epoch. Thereafter very low decrement.</p>
<p>While the value of custom loss decreases almost exponentially/linearly over entire training, starting from 3.2 to -10 (negative) approx.</p>
<p>That means after certain epoch model focuses mainly on custom loss and almost ignores cross entropy loss!</p>
<p>So, what can I do to make the model treat both losses equally and give them dual importance.?</p>
<p>I am currently multiplying a small number to custom loss,but it doesn’t seem to work.</p>

Dwight_Foster: <p>Just an idea but maybe you could clip the custom loss so eventually it stops going down. Then when the model hits that point it might start focusing on the cross entropy loss.</p>

rahulvigneswaran: <p>Hi <a class="mention" href="/u/hdk">@Hdk</a>,</p>
<p>There could be a lot of things at play here depending on your <code>custom_loss</code>. I would typically try the following things,</p>
<ol>
<li>If possible, try to make the custom loss’s lower bound as zero.</li>
<li>If the above is not possible, do <code>Total_loss = (10/2.4)*cross_entropy_loss + custom_loss</code>
</li>
</ol>

Hdk: <p><a class="mention" href="/u/dwight_foster">@Dwight_Foster</a> As I mentioned, I want the model to treat both losses equally i.e. both should go down ideally. So, I don’t want to discard custom loss at some point of time.</p>
<p><a class="mention" href="/u/rahulvigneswaran">@rahulvigneswaran</a><br>
First point is not possible, custom loss can be negative. I already tried second one, but then custom loss stopped decreasing i.e. over entire training I saw almost flat line of loss, while cross entropy was indeed decreasing. It made things even worse. So, I guess it was focusing on just cross entropy loss.</p>
<p>I was thinking to implement a dynamic multiplier instead of static which will be multiplied with custom loss i.e. this multiplier will also change over entire training.<br>
Any idea if that makes sense, how to decide which equation (e.g. ax**2+b…) should that multiplier follow and how it can be implemented in training loop?</p>
<p>Thank you for the reply.</p>
Register_buffer cannot update

chengleo85: <h1>Example:</h1>
<h1>class MyModule(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(QLIF_Conv2d, self).<strong>init</strong>()<br>
self.my_buffer_tmp = torch.zeros(1)<br>
self.register_buffer(‘my_buffer’,my_buffer_tmp)<br>
def forward(self):<br>
self.my_buffer = tensor.ones(1)<br>
…<br>
return …</h1>
<p>I cannot get my_buffer=torch.tensor(1);<br>
I get my_buffer=torch.tensor(1)</p>
<p>How to update my_buffer when forward?</p>

ptrblck: <p>Your code works fine after fixing the <code>tensor.ones</code> to <code>torch.ones</code> and updates the buffer:</p>
<pre><code class="lang-python">class MyModule(nn.Module):
    def __init__(self):
        super(MyModule, self).__init__()
        self.register_buffer('my_buffer', torch.zeros(1))

    def forward(self):
        self.my_buffer = torch.ones(1)
        return

model = MyModule()
print(model.my_buffer)
&gt; tensor([0.])
model()
print(model.my_buffer)
&gt; tensor([1.])
</code></pre>
<p>PS: you can post code snippets by wrapping them into three backticks ```, which makes debugging easier <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Pass raw images without ToTensor()

Kapil_Rana: <p>Hello I want to pass raw images(0-255) to my CNN.</p>
<p>I am not using ToTensor().</p>
<p>img1 = self.loader(os.path.join(self.root1, path1))<br>
img1 = torch.from_numpy(np.asarray(img1)).float()</p>
<p>I got a warning.<br>
UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program.</p>
<p>What it mean, does it will affect CNN in any manner?</p>

ptrblck: <p>No, it shouldn’t affect the model training as long as you don’t try to write to the underlying numpy array.<br>
You could create a copy of the numpy array to get rid of this warning, since <code>from_numpy</code> will share the same data between the tensor and <code>np.array</code>.</p>

Kapil_Rana: <p>I did this</p>
<p>img1 = self.loader(os.path.join(self.root1, path1))<br>
img1 = np.asarray(img1)<br>
img1 = torch.from_numpy((img1)).float()</p>
<p>Still got the warning</p>

ptrblck: <p><code>np.asarray</code> is also using the reference, if no copy is needed.<br>
From the <a href="https://numpy.org/doc/stable/reference/generated/numpy.asarray.html">docs</a>:</p>
<blockquote>
<p>Array interpretation of <em>a</em> . No copy is performed if the input is already an ndarray with matching dtype and order. If <em>a</em> is a subclass of ndarray, a base class ndarray is returned.</p>
</blockquote>

Eta_C: <pre><code class="lang-python">img1 = self.loader(os.path.join(self.root1, path1))
img1 = torch.from_numpy(1.0 * np.asarray(img1)).float()
</code></pre>
How to use MaxUnpool2D after AdaptiveAvgPool2D?

CCLDArjun: <p>Is there a way to get indices from AdaptiveAvgPool2D output so I can use it for the MaxUnpool2D? I looked at the docs and I don’t think it says how to do this.</p>
<p>Manually calculating finding them right after the AdaptiveAvgPool2d seems inefficient.</p>

ptrblck: <p>I’m unsure what indices <code>AdaptiveAvgPool2d</code> should return, so could you explain your use case a bit?<br>
Maxpooling payers are selecting the max. value in the kernel/window using an index and can thus return these indices so that they can be reused in e.g. <code>MaxUnpool2d</code> layers.<br>
However, Avgpool layers are calculating the average in each window, so there is no “selection” involving indices.</p>

Eta_C: <p>Maybe set <code>return_indices=True</code>…</p>

CCLDArjun: <p>Vgg16 has <code>AdaptiveAvgPool2d</code> as its final layer. I’am using vgg’s initial layers to extract features, After which, I want to <code>MaxUnpool2d</code> the output.</p>

ptrblck: <p><code>nn.MaxUnpool2d</code> expects to get indices, which are created by e.g. <code>nn.MaxPool2d</code>.<br>
As described before, neither <a href="https://pytorch.org/docs/stable/generated/torch.nn.AdaptiveAvgPool2d.htmlhighlight=adaptiveavgpool2d#torch.nn.AdaptiveAvgPool2d"><code>nn.AdaptiveAvgPool2d</code></a> nor <a href="https://pytorch.org/docs/stable/generated/torch.nn.AvgPool2d.html?highlight=avgpool2d#torch.nn.AvgPool2d"><code>nn.AvgPool2d</code></a> return indices and setting <code>return_indices=True</code> in these layers will return a <code>TypeError</code>:</p>
<pre><code class="lang-python">nn.AdaptiveAvgPool2d(2, return_indices=True)
&gt; TypeError: __init__() got an unexpected keyword argument 'return_indices'
nn.AvgPool2d(2, return_indices=True)
&gt; TypeError: __init__() got an unexpected keyword argument 'return_indices'
</code></pre>

CCLDArjun: <p>shall I construct my own indices tensor then? Or Is there a way to just call some pytorch function to make that?</p>

ptrblck: <p>You would have to define what these indices should represent.<br>
In max pooling layers the indices are representing the indices of the max. values, which were “selected”.<br>
In avg pooling layers no indexing/selection is done, since the values covered by the filter kernel are averaged, so all values are used.</p>

CCLDArjun: <p>Oops, meant to reply to this one.</p>
<p>Anyways, I want the indices to represent the max values in a 2x2 window with stride = 2. Any way to make the indices quickly?</p>

ptrblck: <p>Yes, you can use <code>nn.MaxPool2d</code> with a <code>kernel_size</code> of <code>2</code> and using <code>return_indices=True</code>.</p>

CCLDArjun: <p>Yeah, but I don’t actually want to maxpool then unpool the feature map, I just want to unpool it</p>
How to define some classes as don't care for time series classification?

Noam_Salomonski: <p>I have a time series classification task in which I should output a classification of 3 classes for every time stamp t.</p>
<p>All data is labeled per frame.</p>
<p>In the data set are more than 3 classes [which are also imbalanced].</p>
<p>I want to use <a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html" rel="noopener nofollow ugc">CrossEntropyLoss</a>.</p>
<p>My net should see all samples sequentially, because it uses that for historical information.<br>
Thus, I can’t just eliminate all irrelevant class samples at preprocessing time.</p>
<p>In case of a prediction on a frame which is labeled differently than those 3 classes, I don’t care about the result.</p>
<hr>
<p>My model outputs a vector of 3 log probabilities, but the label batch has more than 3 labels. Using the <code>weight=</code> parameter with 0 weights for classes I want to ignore yields</p>
<p>{RuntimeError}weight tensor should be defined either for all 3 classes or no classes but got weight tensor of shape: [5] at C:/w/b/windows/pytorch/aten/src\THCUNN/generic/ClassNLLCriterion.cu:44</p>
<p>because the <code>weight</code>'s shape was larger than that of the model output.</p>
<hr>
<p>How to do this correctly in Pytorch?</p>

ptrblck: <p>You could move all classes, which should be ignores to a fixed class index and pass it as <code>ignore_index</code> to <code>nn.CrossEntropyLoss</code>:</p>
<pre><code class="lang-python">batch_size = 100
nb_classes = 3
output = torch.randn(batch_size, nb_classes, requires_grad=True)
target = torch.randint(0, nb_classes+10, (batch_size,))
# move all unwanted classes to "ignore_index"
target[target&gt;=3] = 3

criterion = nn.CrossEntropyLoss(ignore_index=3)
loss = criterion(output, target)
loss.backward()
</code></pre>
How to properly use "einsum" ? It seems to operate faster in some settings

GabbyChan: <p>Hi! Just recently I have tried to use <code>einsum</code> …  I have come to notice that one of the following  forms of sum-product is actually being processed faster?</p>
<p>My matrix multiplication is done in order to achieve the following</p>
<ol>
<li>
<p>I want to calculate the matrix multiplication: B = A^T * A where A^T is NxHxD…  and A = DxHxN… I hope to get the result B in the form of B = NxNxH.</p>
</li>
<li>
<p>Then, I will select the feature element of size 1xH from the upper triangular elements on the NxN<br>
coordinates of B.</p>
</li>
</ol>
<ul>
<li>The question is why the following 1st approach is faster when N becomes larger than 100? when actually the 2nd approach has less number of flops. Did I misunderstand anything ?</li>
</ul>
<p>The 1st approach, which is quite straight forward and is the faster approach, is as follows:</p>
<pre><code class="lang-auto">A_      = [DxHxN]

B1 =  torch.einsum('dhn,dhm-&gt;hnm', A_, A_)     # B1  = [HxNxN]

edge_index_j, edge_index_i = edge_index #  edge_index = [2x #Edges]] 
# where #Edges=N^2/2

B1 = B1[:,edge_index_j,edge_index_i]     # B1  = [Hx#Edges]
B1 = B1.permute(1, 0)                    # B1  = [#EdgesxH]  
</code></pre>
<p>The 2nd slower approach is</p>
<pre><code class="lang-auto">A   = A.permute(2,1,0)                         # A  = [NxHxD]
A_i = torch.index_select(A, 0, edge_index_i)   # A_i = [#EdgexHxD]
A_j = torch.index_select(A, 0, edge_index_j)   # A_j = [#EdgexHxD]

B2 = torch.einsum("ehd,ehd-&gt;eh", A_i, A_j) # B2 = [#EdgexH]
</code></pre>
<p>As N &gt;= 100, I can measure less runtime from the  1st approach than the 2nd approach (here I set N = 2000).</p>
<pre><code class="lang-auto">Runtime (1st): 1.312180 ms
Runtime (2nd): 32.030568 ms
</code></pre>
<p>Here, I measured the runtime with torch.cuda.Event.</p>
<p>However, from the  1st approach, the calculation of B1 is  NxHxDxN = N^2 X H X D, which is twice more than the 2nd approach.</p>
<p>In the 2nd approach, the number of flops for calculating B2 is <span class="hashtag">#EdgexHxD</span>.  where <span class="hashtag">#Edge</span> = N^2/2…</p>
Combine two pertained models with a layer to adjust tensor size

Helen_R6: <p>Hi,<br>
I am currently working on concatenate two pretrained models (BigGAN128 + Resnet18) together. The idea of how to concatenate them is shown in the drawing. The problem is that the output of BigGAN is a tensor of size[1,3,128,128], while resnet only takes input of size [1,3,224,224]. I’ve searched online and found that I can resize the tensor by using nn.function.interpolate(). My question is how can I add this as a layer in my customized model, or there would be other way than interpolate to do the resize work in torch.nn?<br>
Any help and suggestions would be appreciated.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/7/372b8a106351530a95427fc4d8419e34f0179b6c.png" data-download-href="https://discuss.pytorch.org/uploads/default/372b8a106351530a95427fc4d8419e34f0179b6c" title="WechatIMG2588"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/7/372b8a106351530a95427fc4d8419e34f0179b6c_2_690x330.png" alt="WechatIMG2588" data-base62-sha1="7S3xGUANNhp4EczgUwhbTYCQruY" width="690" height="330" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/7/372b8a106351530a95427fc4d8419e34f0179b6c_2_690x330.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/7/372b8a106351530a95427fc4d8419e34f0179b6c_2_1035x495.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/7/372b8a106351530a95427fc4d8419e34f0179b6c.png 2x" data-dominant-color="F6F7FB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">WechatIMG2588</span><span class="informations">1314×630 54.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
what I’ve done:</p>
<blockquote>
<p>import torchvision.models as models<br>
import torch<br>
import torch.nn as nn<br>
import logging<br>
from pytorch_pretrained_biggan import (BigGAN, one_hot_from_names, truncated_noise_sample,<br>
save_as_images, display_in_terminal)</p>
<p>resnet = models.resnet18(pretrained=True)<br>
<span class="hashtag-raw">#load</span> the generator of pretrained biggan128<br>
logging.basicConfig(level=logging.INFO)<br>
<span class="hashtag-raw">#Load</span> pre-trained model tokenizer (vocabulary)<br>
GAN_model = BigGAN.from_pretrained(‘biggan-deep-128’)<br>
final_model=nn.Sequential(GAN_model,resnet)</p>
</blockquote>

Dwight_Foster: <p>You can use interpolate but because it is a pytorch functional function you cannot put it into a sequential model. Instead you could use nn.upsample which will do a similar thing. You can do something like this</p>
<pre><code class="lang-auto">final_model=nn.Sequential(GAN_model, nn.Upsample(size=(224,224)), resnet)
</code></pre>

Helen_R6: <p>Thanks Dwight, your answer truly helped me!!!. Wish you the best!</p>
Problems Generating C++ project with Pytorch and CMake with multiple sourcefiles

jc_dex: <p>I have a file structure like this:</p>
<pre><code class="lang-auto">proj/ 
    includes/ 
        foo.h 
    src/ 
        foo.cpp 
    main.cpp
</code></pre>
<p>Here is my CMakeLists.txt for main.cpp:</p>
<pre><code class="lang-auto">    cmake_minimum_required(VERSION 3.18)
    project(proj)

    set(Torch_DIR /path/to/cmake/Torch)
    find_package(Torch REQUIRED)
    
    include_directories(${PROJECT_SOURCE_DIR}/includes ${TORCH_INCLUDE_DIR})
    add_subdirectory(includes)
    add_subdirectory(src)

    set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${TORCH_CXX_FLAGS}")

    include_directories(includes) 
    include_directories(${OpenCV_INCLUDE_DIRS})
    include_directories(${TORCH_INCLUDE_DIR})

    add_executable(proj main.cpp)
    target_link_libraries(proj foo_lib ${OpenCV_LIBS} "${TORCH_LIBRARIES}")
    set_property(TARGET proj PROPERTY CXX_STANDARD 14)
</code></pre>
<p>Here is my CMakeLists.txt for includes:</p>
<pre><code class="lang-auto">    set(Torch_DIR /path/to/cmake/Torch)
    find_package(Torch REQUIRED)
    include_directories(${PROJECT_SOURCE_DIR}/includes ${TORCH_INCLUDE_DIR})
</code></pre>
<p>Here is my CMakeLists.txt for src:</p>
<pre><code class="lang-auto">    include_directories(${PROJECT_SOURCE_DIR}/includes)
    add_library(foo_lib foo.cpp)
</code></pre>
<p>Upon executing make I get:</p>
<pre><code class="lang-auto">    fatal error: torch/torch.h: No such file or directory
    1 | #include &lt;torch/torch.h&gt;
</code></pre>
<p>But it can link in main.cpp. So obviously my CMakeLists are not created correctly, so any advice on how to create these properly would be appreciated.</p>
Indexing with mapping tensor

convolutions: <p>Hello,</p>
<p>I have the following two tensors:</p>
<ul>
<li>
<p>input RGB image of shape (h,w,c) e.g. (224,224,3)</p>
</li>
<li>
<p>Mapping of shape (h,w, 2) e.g. (224, 224, 2) which provides for every pixel of the image coordinates (x,y)</p>
</li>
</ul>
<p>I want to create now a new output image tensor that contains on the indexes (x,y) the value of the input image. In other words the output should contain on the positions, given by the mapping tensor, the color values of the input image.</p>
<p>How would one do the indexing?</p>
How to avoild for loop while using torch.nn.CosineSimilarity()?

aswamy: <p>I have two tensors, shapes are shown below.</p>
<pre><code class="lang-auto">input1 = torch.randn(1, 3, 10, 8) 
input2 = torch.randn(1, 1, 128, 8)
</code></pre>
<p>I would like to do following operation:</p>
<pre><code class="lang-auto">cos = nn.CosineSimilarity(dim=2, eps=1e-6)
output = cos(input1, input2)
</code></pre>
<p>But I get an error saying that dim 2  has to be the same</p>
<p>Therefore, I had to use <code>for</code> loop as an alternative solution as below</p>
<pre><code class="lang-auto">cos = nn.CosineSimilarity(dim=2, eps=1e-6)
output = []
for idx in range(0,10):
     output.append(cos(input1.[:, :, idx, :].unsqueeze(0).unsqueeze(1), input2))
</code></pre>
<p>This solution works fine. But is it possible to do the same operation without using <code>for</code> loop?</p>
<p>Any hints are helpful.</p>
<p>Thanks!</p>

Valerio_Biscione: <p><code>output = [cos(input1.[:, :, idx, :].unsqueeze(0).unsqueeze(1), input2)) for idx in range(10)]</code> should work.</p>
<p>However I am not sure it makes sense to compute the cosine similarity of a subset of the layer.</p>

aswamy: <p>Thanks for response. Your solution is still the same as mine, right? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> just you use list comprehension.<br>
Cosine similarity is not for layer outputs, but just for two different vectors.</p>

rahulvigneswaran: <p>Can you say what is the output shape that you want?</p>

aswamy: <pre><code class="lang-auto">input1 = torch.randn(1, 3, 10, 8) 
input2 = torch.randn(1, 1, 128, 8)
cos = nn.CosineSimilarity(dim=2, eps=1e-6)
output = cos(input1, input2)
</code></pre>
<p>Expected output shape is <code>(1, 3, 10*128, 1) =&gt;  (1, 3, 1280, 1)</code></p>

aswamy: <p>Its like computing cosine similarity for pairwise vectors of inputs</p>

rahulvigneswaran: <p><a class="mention" href="/u/aswamy">@aswamy</a><br>
``<br>
in1 = input1.squeeze()<br>
in2 = input2.squeeze()</p>
<p>in11 = F.normalize(in1, dim=1)<br>
in22 = F.normalize(in2, dim=0)</p>
<p>torch.cdist(in11, in22, p=2)</p>
<pre><code class="lang-auto">
The output shape would be `(3,10,128)` where the `(10, 128)` would be your 1280 cosine similarity values that you need. Unsqueeze appropriate dims to get the desired shape.

The logic behind this is that,

 `l2_distance(l2_normalized_vector_1, l2_normalized_vector_2) = k*CosineSimilarity(UnNormalized_vector_1, UnNormalized_vector_2)`

where `k` is a constant.

Refer [this](https://stats.stackexchange.com/questions/146221/is-cosine-similarity-identical-to-l2-normalized-euclidean-distance) for more on it.</code></pre>