akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Gradient accumulation for contrastive loss

Guru_Swaroop: <p>I am trying to implement gradient accumulation for large batch training. In the case of categorical cross-entropy loss, I would implement gradient accumulation in the following way:</p>
<pre><code>criterion = cross_entropy_loss
accumulation_steps = 5
model.train()

for idx, (x, y) in enumerate(data_loader, 1):
    output = model(x)
    loss = criterion(output, y) / accumulation_steps
    loss.backward()

    if idx % accumulation_steps == 0:
        optimizer.step()
        optimizer.zero_grad()
</code></pre>
<p>I am unable to use the same strategy using contrastive loss. The contrastive loss for a sample depends on all the samples for that mini-batch. In the context of accumulated mini-batches, the loss for a single sample depends on the outputs from all the samples obtained across the accumulated mini-batches. This seems to be non-trivial for me.</p>
<ul>
<li>One option is to save all the outputs <code>output = model(x)</code> and perform <code>loss.backward()</code> once every <code>accumulation_steps</code>.  However, this would lead to memory issues on GPUs.</li>
</ul>
<p>I wonder, how accumulation can be performed in such a scenario? Looking forward to your assistance. Thanks in advance!</p>
Custom dataset for unsupervised segmentation

Najeh_Nafti: <p>for unsupervised anomaly detection at pixel level, train dataset should contain only normal data and test data should include both normal and anomaly data with their masks.</p>
<pre><code class="lang-auto">
* Train--&gt; Normal

* Test:
   - Normal
   - Not normal

* ground_truth of test data 
</code></pre>
<p>How can I define a custom dataset for that and process it for training and evaluation?<br>
Can I define a dataloader for train images and a custom dataset for test (images+ground_truth)?</p>
Mean of 2d tensor using indices tensor

Ameen_Ali: <p>Hello,<br>
Let’s assume I have a Tensor of shape [M , D] where M is the number of samples and D is the feature dimension. additionally a mask tensor of shape [M , K] which contains K different indices for each of the M samples.</p>
<p>my question is how can I for each of the M samples calculate the mean of its K indices (for each sample we have in the mask which K indices we need to use)</p>
<p>Thanks!</p>

ptrblck: <p>This should probably work, if I understand the use case correctly:</p>
<pre><code class="lang-python">M, D, K = 2, 4, 3
x = torch.randn(M, D)
idx = torch.randint(0, D, (M, K))
x[torch.arange(x.size(0)).unsqueeze(1), idx].mean(1)
</code></pre>

Ameen_Ali: <p>Thanks for your reply!<br>
but notice, in your code you are using the mask for getting features entries, my goal is different<br>
for instance if the first row of the mask contains : [0 , 10 , 13 , 200] I would like to mean X[0 , :] , X[10 , :]  , X[13 , : ] , X[200 , : ] and not X[0 , 0] , X[0 , 10] , X[0 , 13] , X[0 , 200]<br>
Thanks!</p>

ptrblck: <p>Ah OK, so <code>idx</code> would be used in <code>dim0</code>:</p>
<pre><code class="lang-python">M, D, K = 5, 4, 2
x = torch.randn(M, D)
print(x)
&gt; tensor([[ 0.3545, -1.2674, -0.1321, -0.4732],
          [-0.9573,  0.8507, -0.1691,  0.6376],
          [ 0.1386, -1.4488,  0.5860, -0.0135],
          [-0.9486, -0.2253, -1.0952, -0.0432],
          [-0.0213,  0.2111,  0.1952, -0.8802]])

idx = torch.randint(0, M, (M, K))
print(idx)
&gt; tensor([[0, 2],
          [4, 1],
          [1, 2],
          [1, 3],
          [3, 0]])

tmp = x[idx]
print(tmp)
&gt; tensor([[[ 0.3545, -1.2674, -0.1321, -0.4732], # corresponds to first row in idx: [0, 2]
           [ 0.1386, -1.4488,  0.5860, -0.0135]],

          [[-0.0213,  0.2111,  0.1952, -0.8802],
           [-0.9573,  0.8507, -0.1691,  0.6376]],

          [[-0.9573,  0.8507, -0.1691,  0.6376],
           [ 0.1386, -1.4488,  0.5860, -0.0135]],

          [[-0.9573,  0.8507, -0.1691,  0.6376],
           [-0.9486, -0.2253, -1.0952, -0.0432]],

          [[-0.9486, -0.2253, -1.0952, -0.0432],
           [ 0.3545, -1.2674, -0.1321, -0.4732]]])

print(tmp.mean(dim=[1, 2]))
&gt; tensor([-0.2820, -0.0167, -0.0470, -0.2438, -0.4788])
</code></pre>
<p>In this example the first row of <code>idx</code> is <code>[0, 2]</code>, which is then creating <code>[x[0], x[2]]</code>. The last line of code is then taking the <code>mean</code> in <code>dim1</code> and <code>dim2</code>.</p>
Not implemented error of SpatialGridSamplerBilinear on google collab.There is an error in importing pytorch/aten/src/THCUNN/SpatialConvolutionMM.cu possibly

Akshit_Sharma: <p>error:<br>
backend = thnn.type2backend[kernel.type()]<br>
—&gt;          backend.SpatialGridSamplerBilinear_updateOutput(backend.library_state, kernel, grid, output, 0)</p>
<p>/usr/local/lib/python3.6/dist-packages/torch/_thnn/utils.py<br>
in <strong>getattr</strong>(self, name)<br>
25         method = self.methods.get(name, None)<br>
26         if method is None:<br>
—&gt;27           raise NotImplementedError<br>
28         return method</p>
<p><strong>NotImplementedError:</strong></p>
<p>I am working on google colab . I am getting this not implemented issue for <strong>SpatialGridSamplerBilinear_updateOutput</strong>  function.Which is present possibly as <strong>SpatialGridSamplerBilinear.cu</strong> file.</p>
<p>I am working with GPU environment on collab with cuda version : 10.0</p>
<p><strong>!/usr/local/cuda/bin/nvcc --version</strong><br>
nvcc: NVIDIA ® Cuda compiler driver<br>
Copyright © 2005-2018 NVIDIA Corporation<br>
Built on Sat_Aug_25_21:08:01_CDT_2018<br>
Cuda compilation tools, release 10.0, V10.0.130</p>
<p>on collab:<br>
pytorch : 1.1.0<br>
torchvision : 0.3.0<br>
python : 3.6.4</p>
<p>Also if use pytorch 0.4.1 then torchvision 0.3.0 is incompatible.<br>
The same code shows no errors in a perfect nvidia environment in a lab (whose access is rare)<br>
and there pytorch is 0.4.1 and in lab there is a sure call to Spatialsamplerbilinear.cu.</p>
<p>I think it is present in THCUNN and in need to somehow incorporate cudnn in collab environment.</p>
<p>can someone help please…</p>

ttoosi: <p>Did you find a workaround? Same problem here, latest Pytorch throws NotImplementedError for a similar function.</p>

EricPengShuai: <p>Did you solve it now? I meet this problem recently.</p>
Custom dataset with C++ backend

kboroszko: <p>Hi all,</p>
<p>How would one go about extending the basic dataset and writing a backend for it in C++?</p>
<p>I have read the <a href="https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md#contributing-to-pytorch" rel="noopener nofollow ugc">Contribution Guide</a>. It is suggested there that new Ops should be added to ATen native package:</p>
<blockquote>
<p>Modern implementations of operators. If you want to write a new operator, here is where it should go. Most CPU operators go in the top level directory, except for operators which need to be compiled specially; see cpu below.</p>
</blockquote>
<p>However, I don’t want my backend to be a method on a Tensor or a Variable. I would like to be able to create more than just a function. I would like to connect to a database using api in C++ and use that to create a python DataSet object.</p>
<p>For that I would need more than just a function. I would like to be able to create a c++ object and call it from python code. Any ideas how I could do that?</p>
<p>I would also like it to be a part of the pytorch project and not a cusom extention like from <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" rel="noopener nofollow ugc">this tutorial</a>. Any suggestions where would be the best place to put it?</p>

ejguan: <p>I think you want to add a python module using C++. <a href="https://docs.python.org/3/extending/extending.html#extending-python-with-c-or-c" class="inline-onebox" rel="noopener nofollow ugc">1. Extending Python with C or C++ — Python 3.9.6 documentation</a><br>
Or pybind would be a great package to use.</p>
<p>Then you can import the custom module and use it within Dataset</p>

kboroszko: <p>Yeah, thanks for the suggestion. Although, as I mentioned, I would like it to be a part of pytorch package and fit within the structure there.</p>
Performance of a Quantization Aware Trained model without converting

pritom-kun: <p>Hello, this is my first post.</p>
<p>I wanted to know If I want to perform quantization aware training (QAT) and build the model with the torch.nn.intrinsic modules, will the performance of the model drop even if I don’t convert it to a quantized model?<br>
Basically I want to build a model that can achieve it’s best possible performance on a GPU and can be converted into a quantized model whenever needed.</p>
<p>Thanks in advance!</p>

supriyar: <p>Hi <a class="mention" href="/u/pritom-kun">@pritom-kun</a><br>
Can you clarify what you mean by performance of the model here? Are you referring to model numerics or the training time due to enabling QAT on the model?</p>
<p>Regarding the numerics, the QAT step does alter the model weights assuming the model will be quantized at a later stage, so there may be differences if we compare the result with an FP32 trained model.</p>

pritom-kun: <p>That’s what I wanted to know, by the performance I was referring to the prediction accuracy. Thanks for your answer!</p>
PyTorch training crash

vanduong0504: <p>I trained my model with image size 512x512 and GPU RTX2080 11GB. It has been trained 7 epochs and crash. I don’t know the reason for this problem. If it doesn’t have enough VRAM, then it will catch an error at the beginning. Here is my <code>nvidia-smi</code> checking at the beginning:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/6/16e3ab5562bb14c6f7d7b4af5f611db22e74dae9.png" data-download-href="https://discuss.pytorch.org/uploads/default/16e3ab5562bb14c6f7d7b4af5f611db22e74dae9" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/6/16e3ab5562bb14c6f7d7b4af5f611db22e74dae9_2_690x90.png" alt="image" data-base62-sha1="3guhzPv11g917J9ACXN1tmRR6BP" width="690" height="90" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/6/16e3ab5562bb14c6f7d7b4af5f611db22e74dae9_2_690x90.png, https://discuss.pytorch.org/uploads/default/original/3X/1/6/16e3ab5562bb14c6f7d7b4af5f611db22e74dae9.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/6/16e3ab5562bb14c6f7d7b4af5f611db22e74dae9.png 2x" data-dominant-color="2B2B2B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">926×121 33.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Multiple Backwards Calls

explorer90: <p>Hello.  I want to follow the general (simplified) pseudocode structure for training my model:</p>
<pre><code class="lang-auto"># below is all simplified code for one epoch
model.zero_grad()
for sample, target in dataset: # note just one sample
    out = model(sample)
    loss = criterion(out, target)
    loss.backward() # accumulate gradients

# I turned on no grad here to be safe
for param in model.parameters():
    param.copy_(param - lr * param.grad / total_samples) 
</code></pre>
<p>However, I noticed that the model is simply always collapsing to predict the same value for everything. I have confirmed that I can learn something useful if I increase the batch size and manually perform the updates after every backward call but I am doing an expirement where I am trying to accumulate the gradients across every sample.</p>
<p>Is this not a valid approach? Am I doing something incorrectly from the pytorch side of things? Thanks!</p>

ptrblck: <p>The code looks generally alright. Since the updates per iteration seem to work, could you run a test by eg. reusing the same sample 10x in the gradient accumulation loop and check the parameter update afterwards. Based on your code snippet the update should be equal to a single update step (without gradient accumulation).</p>
How to register hook twice?

Sergio814: <p>Hi,</p>
<p>I am trying to visualize the attention maps of my model. There are two pathways that compute attention maps for the visual features (v1 and v2) of two different images, and the features (q) of a question but the attention mechanism is shared. So I have something like this in the forward method of my model:</p>
<pre><code class="lang-auto">v1 = self.attention_mechanism(v1, q) 
v2 = self.attention_mechanism(v2, q)
</code></pre>
<p>Because the attention mechanism is shared by both pathways, registering the hook for, say, attention_mechanism.conv2 will give me the attention map for v2. But I need to register it for both v1 and v2. What would be the way to do it? I tried several things but none of them worked.</p>
<p>Thank you</p>

ptrblck: <p>You could e.g. <code>append</code> the forward activations to a <code>list</code> for the activation (or use any other workflow to store both activations):</p>
<pre><code class="lang-python">activation = {}
def get_activation(name):
    def hook(model, input, output):
        if name in activation:
            activation[name].append(output.detach())
        else:
            activation[name] = [output.detach()]
    return hook


lin = nn.Linear(1, 1)
lin.register_forward_hook(get_activation('lin'))

out = lin(torch.randn(1, 1))
out = lin(torch.randn(1, 1))

print(activation)
&gt; {'lin': [tensor([[0.4515]]), tensor([[0.5995]])]}
</code></pre>
GPU memory allocation with inference_mode(True)

damca: <p>I have trained a Unet-esque model on 512x512 images. Training and evaluation perform without error.</p>
<p>However, I would like to evaluate the model on a much larger image, ~15,000x40,000.</p>
<p>I have 24 GB of total GPU memory. After loading the model’s state dictionary and sending to the gpu,  the gpu memory usage is</p>
<pre><code class="lang-auto">torch.cuda.memory_reserved(0)/1e9
&gt;&gt; 0.24
</code></pre>
<p>After sending the “bigimage” to the gpu, the total GPU usage is ~7.5 GB. Which is expected given the 3 channel <code>float32</code> image.</p>
<p>Then, I try to evaluate the model as follows:</p>
<pre><code class="lang-auto">with torch.inference_mode(True):
    imt = torch.transforms.ToTensor(im)[None]
    imc = imt.to(device)
    out = model(imc)
</code></pre>
<p>Which throws the following OOM error:</p>
<pre><code class="lang-auto">RuntimeError: CUDA out of memory. Tried to allocate 148.56 GiB (GPU 0; 23.68 GiB total capacity; 7.07 GiB already allocated; 14.36 GiB free; 7.08 GiB reserved in total by PyTorch)
</code></pre>
<p>My question is, why is so much memory being allocated while within <code>inference_mode</code>?</p>

ptrblck: <p>The forward pass would calculate the forward activations, which would also need to be stored temporarily, so the OOM issue could be expected. What was the peak memory usage using the <code>512x512</code> images?</p>

damca: <p>Ah yes, that makes sense. Is there a way for the activation tensors to be overwritten or utilized ‘inplace’ (as I write this I am realizing how implausible this is). While training on the 512x512, the peak memory usage is ~6.1 GB.</p>

ptrblck: <p>I’m not sure, if you could save more memory than is already saved by using <code>no_grad()</code> (or the <code>inference_mode()</code>) wrapper. These context managers will make sure to avoid storing forward activations, which would be needed to calculate the gradients, so each forward activation should be deleted after its usage. However, since these activations still have to be computed, you might still run OOM.</p>
How to select the eigenvector of the largest eigenvalues?

nateanl: <p>I have a matrix of dimension (…, N, N). I use <code>torch.linalg.eig</code> to get the eigenvalues and eigenvectors. How to select the eigenvector which has the largest eigenvalue?</p>
<p>Sample code here:</p>
<pre><code class="lang-auto">m = torch.rand(2, 100, 8, 8, dtype=torch.cdouble)
w, v = torch.linalg.eig(m)
_, indices = torch.sort(w.abs(), dim=-1, descending=True)
v_max = f(indices) # the shape of v_max should be (2, 100, 8)
</code></pre>

albanD: <p>Hi,</p>
<p>You can use gather with expanded indices:</p>
<pre><code class="lang-python">import torch
seconds_in_a_day = 24 * 60 * 60
seconds_im = torch.rand(2, 100, 8, 8, dtype=torch.cdouble)
w, v = torch.linalg.eig(seconds_im)
_, indices = torch.max(w.abs(), dim=-1, keepdim=True)
print(indices.size())
print(v.size())
v_max = v.gather(-1, indices.unsqueeze(-1).expand(2, 100, 8, 1)) # expand to 1, 8 if you want rows
print(v_max.size())
print(v[0][0])
print(v_max[0][0])
</code></pre>
No Alias Tensor Operations

river_shah: <p>Eigen and Xtensor have a concept of  <a href="https://eigen.tuxfamily.org/dox/group__TopicAliasing.html" rel="noopener nofollow ugc">no aliasing</a></p>
<p>How do I tell torch to not do any memory allocations when doing tensor multiplication</p>
<pre><code class="lang-auto">torch::Tensor m = torch::rand({size, size});
torch::Tensor v = torch::rand({size, 1});
torch::Tensor r = torch::rand({size, 1});

r = m * v; // r is pre allocated and we should not do memory allocation. how do I tell torch?

</code></pre>

googlebot: <p>general pattern is torch.mul(m,v,out=r), there should be a function with similar signature in c++</p>
<p>that’s about preallocation, I don’t think aliasing per se plays any role like in Eigen, as you’re using higher level wrappers with reference semantics. In particular, inplace ops are explicit (m.mul_(v)) as you’d run into autograd errors otherwise.</p>

river_shah: <p>Thanks for the hint. Matrix multiplication with preallocated tensor can be done like this:</p>
<pre><code class="lang-auto">  torch::Tensor m = torch::rand({size, size});
  torch::Tensor v = torch::rand({size, 1});
  torch::Tensor r = torch::rand({size, 1});
  mm_out(r, m, v);
</code></pre>
Implementing online hard example mining (OHEM)

mjdmahsneh: <p>Hello Pytorchers!! So I have been trying to implement OHEM and have reached to a somehow (I guess) reasonable version. Due to the lack of examples out there, I decided to share this implementation hoping to get an opinion and a fresh eye!</p>
<p>My question comes in two parts:</p>
<ol>
<li>
<p>Is my solution correct? if not, please advise. It would also be great if anyone could direct me towards an optimized solution in terms of generalizability and computation cost.</p>
</li>
<li>
<p>Since in my implementation am only mining the hard negative examples, I was wondering whether this can be done to the positive examples too? In other words, does taking the top-k positive samples to compute the loss for the positive class make sense?</p>
</li>
</ol>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F
import numpy as np

def hard_mining(neg_output, neg_labels, ratio):

    num_inst = neg_output.size(0)
    num_hard = max(int(ratio * num_inst), 1)

    _, idcs = torch.topk(neg_output, min(num_hard, len(neg_output)))

    neg_output = torch.index_select(neg_output, 0, idcs)
    neg_labels = torch.index_select(neg_labels, 0, idcs)

    return neg_output, neg_labels



class BCE_OHEM(nn.Module):
    def __init__(self, ratio):
        super(BCE_OHEM, self).__init__()
        self.ratio = ratio


    def forward(self, inputs, targets):

        pos_idcs = targets[:, 0] &gt;= 0.5

        pos_output = inputs[pos_idcs]
        pos_labels = targets[pos_idcs]

        neg_idcs = targets[:, 0] &lt; 0.5

        neg_output = inputs[:, 0][neg_idcs]
        neg_labels = targets[:, 0][neg_idcs]

        ## locate top-k hard negatives :
        neg_output, neg_labels = hard_mining(neg_output, neg_labels, self.ratio)  # self.num_hard) # *batch_size (we dont do this here, this was useful for detection)
        ## compute loss for negative class over top-k selection :
        neg_BCE_loss = F.binary_cross_entropy(neg_output, neg_labels, reduction='none')

        ## compute loss over positive class :
        pos_BCE_loss = F.binary_cross_entropy(pos_output, pos_labels, reduction='none')

        ## compute overall loss :
        classify_loss = 0.5 * pos_BCE_loss + 0.5 * neg_BCE_loss

        return torch.mean(classify_loss)




if __name__ == '__main__':

    logits = torch.tensor(np.array([[-0.3], [0.8], [-0.4], [0.99]]), requires_grad=True)
    predictions = torch.sigmoid(logits) ## [[0.4256], [0.6900], [0.4013], [0.7291]]
    targets = torch.tensor(np.array([[0.], [0.], [1.], [1.]]), requires_grad=False)

    print(F.binary_cross_entropy(predictions, targets)) ## tensor(0.7386, dtype=torch.float64, grad_fn=&lt;BinaryCrossEntropyBackward&gt;)

    ohem_BCE = BCE_OHEM(ratio=1.0)
    print(ohem_BCE(predictions, targets)) ## tensor(0.7386, dtype=torch.float64, grad_fn=&lt;MeanBackward0&gt;)

    ohem_BCE = BCE_OHEM(ratio=0.5)
    print(ohem_BCE(predictions, targets)) ## tensor(0.8928, dtype=torch.float64, grad_fn=&lt;MeanBackward0&gt;)

    ohem_BCE(predictions, targets).backward()
</code></pre>
Use MKL for BLAS CMake in C++ application

river_shah: <p>I would like to use MKL for BLAS operations. What should I set in my CMakeLists please? Do I need to enable any other build flags and or macros? Below see what my project CMakeLists looks like.</p>
<pre><code class="lang-auto">find_package(Torch REQUIRED PATHS /usr/local/libtorch)
set(LINK_LIBRARIES
    ${CONAN_LIBS}
    ${BLAS_LIBRARIES}
    ${LAPACK_LIBRARIES}
    ${TORCH_LIBRARIES})

Calling C++ code looks like this:

#include &lt;benchmark/benchmark.h&gt;

#include &lt;torch/torch.h&gt;

static void torch_tensor_mult_test(benchmark::State &amp;state) {
  int size = state.range(0);
  torch::Tensor m = torch::rand({size, size});
  torch::Tensor v = torch::rand({size, 1});
  torch::Tensor r = torch::rand({size, 1});

  for (auto _ : state) {
    r = m * v;
  }
}
BENCHMARK(torch_tensor_mult_test)-&gt;RangeMultiplier(2)-&gt;Range(2, 8 &lt;&lt; 7);

BENCHMARK_MAIN();
</code></pre>
What is the difference between wait and barrier in torch distributed?

yxchng: <p>Does <code>handle.wait()</code> block and synchronize all processes like <code>torch.distributed.barrier()</code>? I have having trouble understanding <code>wait</code> and barrier`. Based on the description in torch documentation, I thought wait would also synchronize? Am I misunderstanding? I need to synchronize the results in all processes before continuing. In particular, what are the differences between the 3 blocks of code below.</p>
<blockquote>
<p>x = func()<br>
handle = torch.distributed.all_reduce(x, op=torch.distributed.ReduceOp.SUM, async_op=True)<br>
handle.wait()</p>
<p>x = func()<br>
handle = torch.distributed.all_reduce(x, op=torch.distributed.ReduceOp.SUM, async_op=True)<br>
handle.wait()<br>
torch.distributed.barrier()</p>
<p>x = func()<br>
torch.distributed.all_reduce(x, op=torch.distributed.ReduceOp.SUM, async_op=False)<br>
torch.distributed.barrier()</p>
</blockquote>

cbalioglu: <p>They are in a sense similar, but serve different purposes. <code>wait()</code> ensures that, once returned, the async operation that its is associated with has completed. This means:</p>
<pre><code class="lang-auto">x = func()
torch.distributed.all_reduce(x, op=torch.distributed.ReduceOp.SUM, async_op=False)
</code></pre>
<p>and</p>
<pre><code class="lang-auto">x = func()
handle = torch.distributed.all_reduce(x, op=torch.distributed.ReduceOp.SUM, async_op=True)
handle.wait()
</code></pre>
<p>are in fact equivalent. The advantage of the async version is that you can have additional logic before you call <code>wait()</code> to overlap communication and computation.</p>
<p>On the other hand <code>barrier()</code> is a standalone collective function. You use it mostly to ensure that all processes in your job reach a certain point in execution (e.g. for coordinating checkpointing). The calls to <code>barrier()</code> in your second and third examples are redundant though, since collective reduce operations also implicitly represent a barrier.</p>
How to do channel selection

Peter_Featherstone: <p>I have a tensor of shape [B,C,H,W].<br>
I want a network which outputs a set of probabilities for each channel, i.e. a tensor of shape [B,C,H,W].<br>
I then want to use the probabilities of each channel to select the best channel for each pixel, and output a tensor of shape [B,1,H,W].</p>
<p>So, this is what i have:</p>
<pre><code class="lang-auto">class ChannelDecision(nn.Module):
    def __init__(self, inc):
        super(ChannelDecision, self).__init__()
        self.net    = nn.Sequential(nn.Conv2d(inc, inc, kernel_size=1, bias=True),
                                    nn.ReLU(inplace=True),
                                    nn.Conv2d(inc, inc, kernel_size=1, bias=True),
                                    nn.Softmax(dim=1))

    def forward(self, x):
        probs       = self.net(x)
        decisions   = torch.argmax(probs, dim=1, keepdim=True)
        y = x[decisions] #This doesn't do what i want to do and requires an ENORMOUS allocation
        return y

x = torch.randn(4,32,128,128)
y = n(x) #make this work please
</code></pre>
<p>How can I do this?</p>

Peter_Featherstone: <p>Ok my work around is to have a soft decision, not a hard decision. So do this:</p>
<pre><code class="lang-auto">class ChannelDecision(nn.Module):
    def __init__(self, inc):
        super(ChannelDecision, self).__init__()
        self.net    = nn.Sequential(nn.Conv2d(inc, inc, kernel_size=1, bias=True),
                                    nn.ReLU(inplace=True),
                                    nn.Conv2d(inc, inc, kernel_size=1, bias=True),
                                    nn.Softmax(dim=1))

    def forward(self, x):
        probs = self.net(x)
        x = x * probs
        x = x.sum(1, keepdim=True)
        return x
</code></pre>
Where and how do pip installs of Pytorch install CUDA

ckyleda: <p>When you install Pytorch via pip, I’m aware that it downloads a precompiled CUDA runtime that works with that version of Torch.</p>
<p>However, I cannot seem to find any information on how this works, or where the downloaded runtimes are placed. In a related issue, my version of Torch was clearly <strong>built</strong> with CUDA, yet doesn’t seem to be able to find any cuda runtimes - how is this possible?</p>
<pre><code class="lang-auto">Collecting environment information...
PyTorch version: 1.5.1
Is debug build: No
CUDA used to build PyTorch: 10.2

OS: Ubuntu 16.04.6 LTS
GCC version: Could not collect
CMake version: Could not collect

Python version: 3.6
Is CUDA available: No
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA

Versions of relevant libraries:
[pip3] numpy==1.19.0
[pip3] torch==1.5.1
[pip3] torchvision==0.2.0

</code></pre>
<p>Any information on how pip + pytorch manage the cuda runtime installation, and where this runtime is stored, would be helpful.</p>
Loading vision transform weights to a model error

Roua_Rouatbi: <p>I’m trying to load pth file of a pre-trained vision transformer to a model. The model has the same architecture used for the vision transformer.<br>
I keep getting these errors:<br>
Missing key(s) in state_dict: “module.FeatureExtraction.ConvNet.0.weight”, “module.FeatureExtraction.ConvNet.0.bias”, “module.FeatureExtraction.ConvNet.3.weight”, “module.FeatureExtraction.ConvNet.3.bias”, “module.FeatureExtraction.ConvNet.6.weight”…<br>
Unexpected key(s) in state_dict: “module.vitstr.cls_token”, “module.vitstr.pos_embed”, “module.vitstr.patch_embed.proj.weight”, “module.vitstr.patch_embed.proj.bias”, “module.vitstr.blocks.0.norm1.weight”, “module.vitstr.blocks.0.norm1.bias”, “module.vitstr.blocks.0.attn.qkv.weight”, “module.vitstr.blocks.0.attn.qkv.bias”, “module.vitstr.blocks.0.attn.proj.weight”, …</p>
<p>my code is:<br>
device = torch.device(‘cuda’ if torch.cuda.is_available() else ‘cpu’)<br>
model = Model(opt)<br>
file_path=’/content/drive/MyDrive/VitSTR/vitstr_tiny_patch16_224_aug.pth’<br>
model = torch.nn.DataParallel(model).to(device)<br>
model.load_state_dict(torch.load(file_path))</p>
<p>The error is in loading state dict</p>

Manuel_Alejandro_Dia: <p>Try loading the model before wrapping it as a <code>DataParallel</code> object:</p>
<pre><code class="lang-auto">device = torch.device(‘cuda’ if torch.cuda.is_available() else ‘cpu’)
model = Model(opt)
file_path=’/content/drive/MyDrive/VitSTR/vitstr_tiny_patch16_224_aug.pth’
model.load_state_dict(torch.load(file_path))
model = torch.nn.DataParallel(model).to(device)
</code></pre>
<p>This is because once you wrap the model in <code>DataParallel</code>, you will need to write <code>model.module</code> to access your original object of class  <code>Model()</code> .</p>
<p>Also, remember that it recommended to always use <code>DistributedDataParallel</code> over <code>DataParallel</code> whenever possible.</p>
<p><a href="https://yangkky.github.io/2019/07/08/distributed-pytorch-tutorial.html" rel="noopener nofollow ugc">This a guide</a> that I found really useful to learn about how to use the <code>DistributedDataParallel</code> for multi-gpu/multi-node training. <img src="https://discuss.pytorch.org/images/emoji/apple/grinning_face_with_smiling_eyes.png?v=10" title=":grinning_face_with_smiling_eyes:" class="emoji" alt=":grinning_face_with_smiling_eyes:"></p>
Cyclic constraints on nn.Modules output

Bawaw: <p>Hello community,</p>
<p>I’m trying to put cyclic constraints on the output of a normalising flow but I am uncertain of how this could be achieved in PyTorch.</p>
<p>Mathematically I would like the flow to satisfy these constraints:</p>
<ul>
<li>f(-π)=-π</li>
<li>f(π)=π</li>
<li>∇f(π) = ∇f(-π)</li>
</ul>
<p>Any idea how this could be done without adding these as soft constraints in the objective?</p>
<p>reference: Rezende, Danilo Jimenez, et al. “Normalizing flows on tori and spheres.” International Conference on Machine Learning. PMLR, 2020.</p>
<p>Thanks</p>
Using torchvision on android

julien-thierry: <p>Hi,</p>
<p>I see there are some android builds for android:<br>
<a href="https://mvnrepository.com/artifact/org.pytorch/pytorch_android_torchvision/1.9.0" class="onebox" target="_blank" rel="noopener nofollow ugc">https://mvnrepository.com/artifact/org.pytorch/pytorch_android_torchvision/1.9.0</a></p>
<p>However, while they do ship libtorchvision.so, they do not ship any header.<br>
How can we register the torchvision operators as described here <a href="https://github.com/pytorch/vision#torchvision-operators" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pytorch/vision: Datasets, Transforms and Models specific to Computer Vision</a>, without the torchvision header?</p>
<p>Thanks,</p>
<p>Julien</p>
IOU metric in sematic segmentation

Sarmad_GTU: <p>Hi all<br>
I want to ask about the IOU metric  for multiclass semantic segmantation<br>
can I use this code from the semantic segmentation PyTorch model  to calculate the IOU</p>
<p>def iou(pr, gt, eps=1e-7, threshold=None, ignore_channels=None):<br>
“”“Calculate Intersection over Union between ground truth and prediction<br>
Args:<br>
pr (torch.Tensor): predicted tensor<br>
gt (torch.Tensor):  ground truth tensor<br>
eps (float): epsilon to avoid zero division<br>
threshold: threshold for outputs binarization<br>
Returns:<br>
float: IoU (Jaccard) score<br>
“””</p>
<pre><code>pr = _threshold(pr, threshold=threshold)
pr, gt = _take_channels(pr, gt, ignore_channels=ignore_channels)

intersection = torch.sum(gt * pr)
union = torch.sum(gt) + torch.sum(pr) - intersection + eps
return (intersection + eps) / union
</code></pre>
<p>my target mask is: 8x512x512<br>
my predicted mask is: 8x5x512x512<br>
where is 5: No of class<br>
8: batch size</p>

KFrank: <p>Hi Sarmad!</p>
<aside class="quote no-group" data-username="Sarmad_GTU" data-post="1" data-topic="129702" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sarmad_gtu/48/41574_2.png" class="avatar"> Sarmad_GTU:</div>
<blockquote>
<p>can I use this code from the semantic segmentation PyTorch model  to calculate the IOU</p>
</blockquote>
</aside>
<p>I haven’t looked at your code (After all, who knows what <code>_threshold()</code><br>
might do?), but if your intersection-over-union function does use a<br>
“threshold for outputs binarization,” it will not be (usefully) differentiable,<br>
and therefore can’t be used as a loss function for backpropagation and<br>
training.  (<code>CrossEntropyLoss</code> is a good loss function to start with for<br>
semantic segmentation.)  IoU (even with thresholding) is, of course, a<br>
perfectly reasonable evaluation metric.</p>
<p>Best.</p>
<p>K. Frank</p>

Sarmad_GTU: <p>thank you<br>
the threshold is to binarize the predicated output I don’t now why they threshold out.<br>
i have 5 class in output so the size is<br>
8x5x512x512<br>
and the gt (ground truth) is 8x512x512<br>
how can i calculate the semantic segmentation metric (IOU and F1 ) ?<br>
any suggestion !</p>
Run time Error in Focal loss of smp.loss

Sarmad_GTU: <p>Hi all<br>
i am new in using of Sematic Segmantation models<br>
so i used the smp modsl for smenatic segamtnion amd i built model and loss lik follosing :</p>
<p>model = smp.Unet(encoder_name=‘resnet34’,<br>
encoder_depth=5,<br>
encoder_weights=‘imagenet’,<br>
decoder_use_batchnorm=True,<br>
decoder_channels=(256, 128, 64, 32, 16),<br>
decoder_attention_type=None,<br>
in_channels=3,<br>
classes=5,<br>
activation=None,<br>
aux_params=None)</p>
<p>loss = smp.losses.FocalLoss(mode=‘multiclass’, gamma=2.0)<br>
loss.<strong>name</strong> = ‘FocalLoss’</p>
<p>and the target mask size is 8x512x512  (contain indices in each pixel represents the class value)<br>
with image size is 8x3x512x512</p>
<p>after run the following code to train the model :</p>
<p>train_epoch = smp.utils.train.TrainEpoch(<br>
model=model,<br>
loss=loss,<br>
metrics= metrics,<br>
optimizer=optimizer,<br>
device=device,<br>
verbose=True,)<br>
train_logs = train_epoch.run(traindataloader)</p>
<p>i got this error:<br>
…/ Deeplearning\lib\site-packages\segmentation_models_pytorch\utils\functional.py", line 34, in iou<br>
intersection = torch.sum(gt * pr)<br>
RuntimeError: The size of tensor a (8) must match the size of tensor b (5) at non-singleton dimension 1</p>
<p>why the gt and pr mismatch ??<br>
how can I overcome this error?</p>

Sarmad_GTU: <p>when I Remove the metric list the error is removed but I can’t show any metric during running.<br>
the error comes from : intersection = torch.sum(gt * pr)<br>
the size of gt is 8x512x512<br>
the size of pr is 8x5x512x512<br>
where the 8 is batch size and 5 is the class number</p>
<p>how can i solve this issu ?</p>

ptrblck: <p>I’m not familiar with the <code>smp</code> repository and their metrics, but based on the error message I would suggest to check the inputs to this particular metric function and check what the expected shapes would be (e.g. you might miss an <code>argmax</code> operation or any other op which changes the shape).</p>

Sarmad_GTU: <p>thank you for replying<br>
actually, I want to calculate the Semantic segmentation metrics like:<br>
MIoU, F1 Score , IOU per class, and Confusion matrix , so i used a already define metric function but it doesn’t get the correct value<br>
I have multiclass (No of the class is 5) and I want to calculate above metric<br>
any suggestion?<br>
thank you in advance</p>
My model's layer's size changed during training

Sung: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/8/c8b6a748feb649dc05f9839ca1a6f1afa0aabfe1.png" data-download-href="https://discuss.pytorch.org/uploads/default/c8b6a748feb649dc05f9839ca1a6f1afa0aabfe1" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/8/c8b6a748feb649dc05f9839ca1a6f1afa0aabfe1.png" alt="image" data-base62-sha1="sDAPid0hZVafSjokZRwWRMdnGwx" width="690" height="136" data-dominant-color="EFEEEE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">958×190 3.54 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I’m currently using 3DefficientNet and When I try to load the state_dict file I have a error like this so I found out why this error came up. And the problem was that the size of the layer have changed during the training. I don’t want this thing to happen… Does any body have solution for this?</p>
How to convert optional<Tensor> to Tensor

lcxywfe: <p>I’m writing a custom op for TorchScript, and the c++ function is like</p>
<pre><code class="lang-auto">at::Tensor myforward(const at::Tensor&amp; input0,  // input
                           const at::Tensor&amp; input1,  // weight
                           const optional&lt;Tensor&gt;&amp; input2,  // bias
</code></pre>
<p>because the bias may be None in python, and how to convert the optional to Tensor?<br>
I tried <code>input2.value_or(Tensor{})</code> but couldn’t be compiled.</p>

tom: <p>You can dereference it (<code>*input2</code>)  if it <code>.has_value()</code> is true).<br>
As far as I can tell, these type of utility interfaces usually designed to be compatible with the usual (here  <a href="https://en.cppreference.com/w/cpp/utility/optionalhttps://en.cppreference.com/w/cpp/utility/optional" rel="noopener nofollow ugc"><code>std::optional</code></a>, interfaces) but is duplicated to avoid implementation quirks with older (?) c++ compilers.</p>
<p>Best regards</p>
<p>Thomas</p>
Loss becoming 'nan' when training my model on another system

Homa: <p>Hi<br>
I have a semantic segmentation model. this model train fine on my system(on  single gpu) but when I am training the same modal and the same parameters on another system  (on single gpu and two gpus with Dataparallel) the output and loss is nan, while version of pytorch and cuda  is the same in two systems. can you help me? thanks</p>
RuntimeError: CUDA unknown error

amahoro12: <p>RuntimeError: CUDA unknown error - this may be due to an incorrectly set up environment, e.g. changing env variable CUDA_VISIBLE_DEVICES after program start. Setting the available devices to be zero.</p>
<p>I need your help on how above issues should be resolved<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/6/069a7f745f04b6bb84e6bbb3cd2f3b6a2c3d987b.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/069a7f745f04b6bb84e6bbb3cd2f3b6a2c3d987b" title="Capture1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/6/069a7f745f04b6bb84e6bbb3cd2f3b6a2c3d987b_2_690x175.jpeg" alt="Capture1" data-base62-sha1="WpSfqys8D1UoCKo3d4Wmxwv9kn" width="690" height="175" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/6/069a7f745f04b6bb84e6bbb3cd2f3b6a2c3d987b_2_690x175.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/0/6/069a7f745f04b6bb84e6bbb3cd2f3b6a2c3d987b_2_1035x262.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/6/069a7f745f04b6bb84e6bbb3cd2f3b6a2c3d987b_2_1380x350.jpeg 2x" data-dominant-color="2F2E2F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture1</span><span class="informations">1817×462 70.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Based on the error message it seems that your setup is unable to initialize the CUDA context, which could be caused by e.g. a faulty driver installation, a driver update without restarting the node, using the “sleep/hibernate” mode in Linux etc.<br>
Was this setup working before at one point or is this a new setup?</p>

amahoro12: <p>Thank you a lot for your help.<br>
My laptop model is Xiaomi Redmibook 16.<br>
I am running above code in pycharm 2021.2, python 3.9.  At the beginning it display no NVIDIA Graphic driver and i install NVIDIA GeForce 610M. It didn’t work before.</p>

ptrblck: <p>The GeForce 610M is a Fermi GPU with compute capability 2.1, so it’s not supported anymore as it’s too old. You could try out e.g. Google Colab with the free GPU runtime instead.</p>

amahoro12: <p>Thank you a  lot. I really appreciate it. let me try it.</p>
UserWarning when using 'same' padding

hassannn_nasrala: <p>Hello, I tried to use a 4x4 convolution with ‘same’ padding like this:</p>
<pre><code class="lang-auto">conv = nn.Conv2d(
                 in_channels = in_channels,
                 out_channels = out_channels,
                 kernel_size = 4,
                 stride =  1,
                 padding =  'same',
                 bias = False
                              )
</code></pre>
<p>I don’t get any errors, and it seems that it works fine since the input and output tensors do have the size…</p>
<p>But I get this UserWarning which I am curious to know what exactly does it mean and wither I should be aware of any problems</p>
<blockquote>
<p>UserWarning: Using padding=‘same’ with even kernel lengths and odd dilation may require a zero-padded copy of the input be created (Triggered internally at /pytorch/aten/src/ATen/native/Convolution.cpp:660.)</p>
</blockquote>
<p>Thank you.</p>

ptrblck: <p>If I understand the warning correctly, it would point to an additional copy of the input tensor (which could downgrade the performance) due to the used setup. If you don’t see any slowdowns in your code (as it might not be triggered or might not even be the bottleneck), I would guess you could ignore it.</p>
Pytorch+cuda version for geforce gt 730

Istvan_Ferenc_Toth: <p>Hi. I have not worked wit GPUs yet, so I am new to this topic. Can someone suggest which pytorch and cuda version are working with a Nvidia Geforce GT 730 GPU  ? I searched for this on the internet, but it is very confusing at first glance. However, I figured out that the  my GPU has 3.5 compute capability (not sure how this relates to the pytorch and cuda version I need). Many thanks!</p>

ptrblck: <p>The current PyTorch pip wheels and conda binaries ship with a min.compute capability of 3.7, so you would need to build PyTorch from source for your GPU as described <a href="https://github.com/pytorch/pytorch#from-source">here</a>. You might be able to use an older release (I believe ~1.5 might have used 3.5 as the min. compute capability, but I’m not sure), but you would miss the latest features and fixes.</p>
Is there any way to obtain the computation graph of an NN?

crissallan: <p>Hi,</p>
<p>I’d like to get the computation graph of a self-defined NN, is there any way in tensorboard that could help me to achieve that?</p>

ptrblck: <p>The <a href="https://pytorch.org/tutorials/intermediate/tensorboard_tutorial.html">Tensorboard Tutorial</a> has a small example on how to visualize the model. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Saving matplotlib.pyplot scatter plot in TensorBoard?

aksg87: <p>Is there a convenient way to save a matplotlib.pyplot scatter plot into TensorBoard using the SummaryWriter?</p>

Syzygianinfern0: <p>You can check this out: <a href="https://github.com/wookayin/tensorflow-plot" class="inline-onebox" rel="noopener nofollow ugc">GitHub - wookayin/tensorflow-plot: 📈 TensorFlow + Matplotlib as TF ops</a></p>

aksg87: <p>I saw that but that requires importing TensorFlow I believe?</p>