akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How can I calcaulte 5% & 95% of of the whole datasets of images?

Ali_Boushehri: <p>Hi</p>
<p>I have a pretty big dataset of images. I would like to calculate the 5% and 95% of pixel ranges for the whole dataset.</p>
<p>I am wondering if there is any method for that? Also, how can I do that in pytorch?</p>
<p>Please let me know if you have any quetions</p>

tom: <p>Hi Ali,</p>
<p>if your values are 8 or 16 bit integers (which is quite common), I’d probably just compute a histogram.<br>
If they are not, you could do this iteratively: Quantize to 8 bits by rounding down and up (keeping statistics for both rounded-down and rounded-up). Then you know the 5% percentile is between the rounded-down 5% and the rounded up 5% percentile and then you can just quantize that range to 8 bits (if you want it easy, just clamp the range) - so everything outside the range gets put on the boundaries, but the 5% quantile is in the “higher resolution range”.</p>
<p>To make things concrete</p>
<pre><code class="lang-python">data = torch.randn(500000, dtype=torch.double)
q05_true = data.sort().values[int(len(data) * 0.05)]  #  if you want data.quantile(0.05), you would have to match their interpolation
# upper and lower bound
q05_max = torch.max(data)
q05_min = torch.min(data)

HIST_SIZE = 1000

done = False
while not done:
    # each loop means a loop over your dataset
    transformed = ((data - q05_min) * HIST_SIZE / (q05_max - q05_min)).clamp(min=0, max=HIST_SIZE)
    ceil = transformed.ceil().long()
    floor = transformed.floor().long()

    vals_c, counts_c = torch.unique(ceil, return_counts=True)
    vals_f, counts_f = torch.unique(floor, return_counts=True)

    # refined upper and lower bound
    q05_max_new = (vals_c[(counts_c.cumsum(-1).double()/data.numel() &gt; 0.05).nonzero().min()]) * (q05_max - q05_min) / HIST_SIZE + q05_min
    q05_min_new = (vals_f[(counts_f.cumsum(-1).double()/data.numel() &lt;= 0.05).nonzero().max()]) * (q05_max - q05_min) / HIST_SIZE + q05_min

    assert q05_min_new &lt;= q05_true &lt;= q05_max_new, f"{q05_min_new} &lt;= {q05_true} &lt;= {q05_max_new}"
    q05_min, q05_max = q05_min_new, q05_max_new
    done = len(vals_c) == 3
    print(f"{q05_min}, {q05_max}")

vals, counts = data.clamp(min=q05_min, max=q05_max).unique(return_counts=True)
q05 = vals[(counts.cumsum(-1) &lt;= int(len(data) * 0.05)+1).nonzero().max()]
print("found", q05.item(), "true", q05_true.item())
</code></pre>
<p>Actually a fun task, thanks for sharing the problem!</p>
<p>Best regards</p>
<p>Thomas</p>
Loss formulation which involves model parameters

Anshul_Tomar: <p>Giving a dummy loss formulation here :-<br>
Let L = A*BCELoss(output, ground_truth) + B where A, B are model parameters.<br>
How would one proceed to define the loss function and model for this loss formulation.</p>

carloalbertobarbano: <p>Is the question about how to make A and B learnable parameters?<br>
In that case, you could add A and B to your model class definition as <code>nn.Parameter</code> and then compute the loss as</p>
<pre><code class="lang-auto">loss = model.A * bce(output, gt) + model.B
</code></pre>
Layer for General Weight Sharing

nick-fung: <p>So I’m trying to implement a layer that performs weight sharing on a per-neuron basis.</p>
<p>I’ll give a basic example: We have 2 scalar parameters, <code>a</code> and <code>b</code>, and we have 3 dimensional inputs and outputs, and we want the weight matrix to be given by:</p>
<pre><code class="lang-auto">a b b
b a b
b b a
</code></pre>
<p>One way I came up with is this:</p>
<pre><code class="lang-auto">class WeightSharing(nn.Module):
    def __init__(self):
        super().__init__()
        shared_weights = torch.nn.Parameter(torch.randn(2))
        self.register_parameter("shared_weights",shared_weights)
        self.index_map = torch.LongTensor([0,0,1, 0,1,0, 1,0,0])
        
    def forward(self, x):
        W = self.shared_weights[self.index_map].view(3, 3)
        return torch.matmul(x, W)
</code></pre>
<p>The problem is that indexing the shared weights on each forward pass seems to be detrimental to performance. Can anyone recommend a better approach to this? Is there a layer that can perform a view based on a fixed index mapping?</p>
Loading big input tensor to GPU on Android

julien-thierry: <p>Hi,</p>
<p>In an Android program, I want to run a model optimized for Vulkan. The main issue I encounter is that my input tensor seem to exceed some limit, I seem to be unable to load a tensor bigger than {1, 4, 2048, 2048}.<br>
The 2048 seem to be the limit maxImageDimension3D of my Vulkan device, however I cannot find hard data on the 4 channel limit.</p>
<p>Here’s the loading code I have</p>
<pre><code class="lang-auto">auto tensor_a = at::rand({1, channels, 2048, 2048}, at::device(at::kCPU).dtype(at::kFloat));
auto tensor_a_vulkan = tensor_a.vulkan();
</code></pre>
<p>If “channels” is anything higher than 4 (or the image sizes is bigger), it will result in:</p>
<pre><code class="lang-auto">terminating with uncaught exception of type c10::Error: VkResult:-2
Exception raised from buffer at &lt;prefix&gt;pytorch/aten/src/ATen/native/vulkan/api/Resource.cpp:479 (most recent call first):
(no backtrace available)
Aborted
</code></pre>
<p>There is still memory available on the device, I can safely allocate more tensors on the CPU and load them to Vulkan with no issues (as long as the individual dimensions are below the apparent limits) .</p>
<p>Is this a known limitation? Is there any way to work around it?</p>
<p>Thanks,</p>
<p>Julien</p>

ssjia: <p>This seems to be a limitation with the Vulkan API, unfortunately there isn’t any workaround for it. Although there may technically be enough memory on the device to store the tensor data, whatever propriety technique used to represent textures on the device probably can’t handle textures of that size. A solution would involve using multiple textures to represent one tensor if the dimensions exceed the limit, which we don’t plan to implement at the moment.</p>
<p>Just curious, what GPU/device are you using?</p>

julien-thierry: <p>Thanks for the reply. However, how does this work for tensors inside a model? Do only models only using tensors smaller than <code>4 x maxImageDimension3D x maxImageDimension3D</code> throughout all their layers work on Vulkan at the moment?</p>
<p>This was tested on a device with Snapdragon S865 (GPU Adreno 650).</p>

julien-thierry: <p>Hi,<br>
Would you have any additional information regarding the internal layers of models loaded to Vulkan GPU with regard to these limits?</p>
<p>Thanks,</p>
<p>Julien</p>
Built-in way to calculate accuracy over padded sequences of softmax/logsoftmax outputs?

josmi9966: <p>Is there some built-in method for calculating prediction accuracy over sequences of softmax/logsoftmax vectors?</p>
<p>So i have a batch of sequences, the inputs padded by some padding index, the targets of identical lengths where the targets are class indices and the prediction of the network is in the form of softmax or logsoftmax scores.</p>
<p>I would like to calculate the accuracy of the predictions on the current batch as efficiently as possible. This means, all the elements in each of the sequences where the input is a padding index need to get ignored and for the  rest, the prediction index needs to get calculated as the index of the element in the output vector with the highest prob or log/prob and then all the correctly predicted indices need to get counted etc.</p>
<p>The loss functions like LogSoftmax automatically take care of ignoring padding indices. Is there an equivalent method for calculating accuracy?</p>
<p>And also, what if the sequences are packed, is there are method for that situation?</p>

Samil: <p>Hey John. Have you had any answers to your question?</p>
How to provide image and number as input to model

joyjeni: <p>I want to create a model which takes two inputs</p>
<ol>
<li>digit image from MNIST dataset</li>
<li>random number from 0-9</li>
</ol>
<p>Output<br>
Sum of two numbers</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/2/82c34900b836a852edbc2509b398db2ed7eadb5b.png" alt="image" data-base62-sha1="iEMsZPTAOYBXbhqVG3RsV8Yt8JZ" width="628" height="235"></p>
<p>How to create this model using pytorch</p>
Iter __getitem__ from within self.__getitem__

ed-fish: <p>I am using <code>__getitem__</code> to grab multiple embeddings from a file system and concat them into a single sample which is then sent to my dataloader for batching.</p>
<p>The dataset is not very well sanitised and occasionally I will need to skip a sample. Is there a way to return and skip a sample, or iterate <code>__getitem__</code> from within the method itself?</p>
<p>Alternatively - should this code be put somewhere else?</p>
<p>Thanks for any help.</p>
<p>Ed <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

the-dharma-bum: <p>Could you provide a bit more informations about your use case ?</p>
<p>Why can’t you identify indices that correspond to corrupted file, for instance during the dataset instanciation, by looping one time over all the files, or even simplier identify those files before the dataset instanciation ?</p>
<p>In your training loop, I assume you’re doing something alike</p>
<pre><code class="lang-python">for batch in dataloader:
    inputs, targets = batch
    outputs = model(inputs)
    ...
</code></pre>
<p>Can’t you detect batches to skip in this loop ?</p>
<p>If for some reasons you can’t do any of these propositions, you have to possibility to define a Sampler object that will be use by a DataLoader object to samples indices. Those indices will then be used to call the  <code>__getitem__</code> method of a Dataset object. Check the <a href="https://pytorch.org/docs/stable/data.html#" rel="noopener nofollow ugc">doc of pytorch.utils.data</a> for more infos. Perhaps in the <code>__next__</code> method of your sampler you could try and detect corrupted files and avoid yielding them so that the <code>__getitem__</code> method will never be called with their corresponding indices.</p>
<p>I hope I’m being clear, feel free to ask for more details of course.</p>

ed-fish: <p>Thank you <a class="mention" href="/u/the-dharma-bum">@the-dharma-bum</a> for your reply.</p>
<p>It seems that a <code>custom sampler</code> is exactly what I require in this case. This will allow me to check that all samples in the <code>[idx] folder</code> have been processed correctly before sending to the <code>dataloader</code>.</p>
<p>This is a great solution for me as the dataset is being processed while I am experimenting with some model architectures. Identifying incompete samples during <code>init</code> wouldn’t be possible due to the dataset size.</p>
<p>Thank you for your help!</p>
Is there a way to convert a local PyTorch model to Torch Scipt Model?

Star_J: <p>Hi, guys,  I’m learning PyTorch and a total beginner.<br>
How do I convert a downloaded PyTorch model (<a href="https://drive.google.com/file/d/1K_xN32uoQKI8XmNYNLTX5gDn1UnQVe5I/view" rel="noopener nofollow ugc">https://drive.google.com/file/d/1K_xN32uoQKI8XmNYNLTX5gDn1UnQVe5I/view</a>) to Torch Scirpt Model. I have tried to use Python packages, torchvision and torch, but I have no idea how to load a model form local path.<br>
Any help would be appreciated.</p>

ptrblck: <p>I don’t know what’s exactly stored in this file, but the common workflow would be to create a model instance and load its <code>state_dict</code> as seen in this code snippet:</p>
<pre><code class="lang-python">model = MyModel()
model.load_state_dict(torch.load(PATH_TO_STATE_DICT))
</code></pre>

Star_J: <p>The code works fine. Thank you very much</p>
Behavior of model.eval(). Does it disable gradient operations?

phantom90: <p>Hi there,</p>
<p>Consider the following circumstance:</p>
<pre><code class="lang-auto">model.eval()
y = model(x)
loss = criterion(y, label)
# backward() and step() the loss, abbreviate.
</code></pre>
<p>I just want to calculate the gradient and update parameters when things like BN and Dropouts are disabled. Are there any potential issues here?</p>
<p>Thanks~</p>

ptrblck: <p><code>model.eval()</code> does not disable the gradient calculations, but only changes the behavior of some layers (such as batchnorm and dropout), so I wouldn’t expect to see any issues with Autograd.</p>
How to save model checkpoints to resume training

talhaanwarch: <p>I am confused how to save model for the purpose of resume training<br>
I have a function</p>
<pre><code class="lang-auto">def fit(model,train,val):
    .....
    .....
    return loss
</code></pre>
<p>I configure model as follow</p>
<pre><code class="lang-auto">model=MyModel.to(device)
criterion = nn.BCEWithLogitsLoss()
opt=torch.optim.AdamW(params=model.parameters(),lr=0.001)
</code></pre>
<p>Then I call function as<br>
<code>loss=fit(model,train,val)</code></p>
<p>Do I need to return optimizer and model from my fit function to save checkpoint as follow</p>
<pre><code class="lang-auto">state = {
        'epoch': epoch,
        'state_dict': model.state_dict(),
        'optimizer': opt.state_dict(),
}
savepath='0to5.pt'
torch.save(state,savepath)
</code></pre>

ptrblck: <p>Your approach looks correct and is also explained in <a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html">this tutorial</a>.</p>

talhaanwarch: <p>sorry, its not clear to me. I am not sure whether model variable out of function is updated or we need to return model object</p>
Inplace addition for l2 regularization

kavka: <p>Suppose I need to add a l2 regularization term to my loss, and its calculated like below</p>
<pre><code class="lang-auto">regularizer = torch.tensor(0.)
for name, param in model.named_parameters():
     if 'weight' in name:
         regularizer += torch.sum(torch.norm(param, dim=0))
</code></pre>
<p>I have two questions<br>
one is that, for the initilization of ‘regularizer’, do I need to set the ‘requires_grad=True’</p>
<pre><code class="lang-auto">regularizer = torch.tensor(0., requires_grad=True)
</code></pre>
<p>Second one is that, is the inplace addition appropriate here? Do I need to modify it to</p>
<pre><code class="lang-auto">regularizer = regularizer  + torch.sum(torch.norm(param, dim=0))
</code></pre>

ptrblck: <ol>
<li>No, you don’t need to use <code>requires_grad</code> attribute and could even create the initial <code>regularizer</code> variable as a Python scalar: <code>regularizer = 0.</code>
</li>
<li>As long as Autograd doesn’t raise an error, the inplace operation should work fine. In case you are concerned about scripting the model (and loss calculation), you could check if removing the inplace ops would allow more operator fusions, but that might be an edge case.</li>
</ol>
Running examples from PyTorch C++ API

PersonalFinance004: <p>I’m trying to get more familiar with ATen by playing around with it directly.</p>
<p>I would like to run the examples from the <a href="https://pytorch.org/cppdocs/" rel="noopener nofollow ugc">cppdocs</a>, for example</p>
<pre><code class="lang-auto">#include &lt;ATen/ATen.h&gt;

at::Tensor a = at::ones({2, 2}, at::kInt);
at::Tensor b = at::randn({2, 2});
auto c = a + b.to(at::kInt);
</code></pre>
<p>I’m not super familiar with C++, so what I did was create a new C++ project <code>my_project</code> such that I get the following folder structure:</p>
<pre><code class="lang-auto">+ pytorch/
|
+ my_project/
</code></pre>
<p>and in <code>my_project/</code> create a <code>main.cpp</code>:</p>
<pre><code class="lang-auto">#include "../pytorch/aten/src/ATen/ATen.h"

int main() {
  at::Tensor a = at::ones({2, 2}, at::kInt);
  at::Tensor b = at::randn({2, 2});
  auto c = a + b.to(at::kInt);
  return 0;
}
</code></pre>
<p>The <code>include</code> seems to be successful in that it finds the folder, however when I try to build <code>my_project</code> I get the following error:</p>
<pre><code class="lang-auto">In file included from .../aten_debug/main.cpp:2:
.../aten_debug/../pytorch/aten/src/ATen/ATen.h:7:10: fatal error: 'c10/core/Allocator.h' file not found`.
</code></pre>
<p>What do I have to do to get this to work? Note please that <code>pytorch</code> isn’t build, it’s just the repository, since I’m working on a laptop. I have the possibility to use a remote server with built PyTorch, but I’m not sure whether this is the problem or just some trivial misunderstanding due to my lacking C++ knowledge.</p>

ptrblck: <p>You could follow <a href="https://pytorch.org/cppdocs/installing.html">this tutorial</a>, which explains how to write the <code>CMakeLists.txt</code>, setup the general project, and build it, as it seems you are trying to use a manual setup, which is missing some includes.</p>
Training via Supervised Learning a Neural Network with Multiple Output Layers

ksivakumar: <p>I have a neural network that has multiple output layers for softmax probability depending on the state my agent is in. Here is an example of my network:</p>
<pre><code>class policy(nn.Module):
   def __init__(self):
      hidden_layer = 32
      super(policy, self).__init__()
      self.affine1 = nn.Linear(3, hidden_layer)
      self.affine2 = nn.Linear(hidden_layer, hidden_layer)
      self.output1 = nn.Linear(hidden_layer, 10)
      self.output2 = nn.Linear(hidden_layer, 5)
      self.output3 = nn.Linear(hidden_layer, 3)

   def forward(self, x):
      x = torch.nn.functional.relu(self.affine1(x))
      x = torch.nn.functional.relu(self.affine2(x))
      outputprobs1 = torch.nn.functional.softmax(self.output1(x), dim=-1)
      outputprobs2 = torch.nn.functional.softmax(self.output2(x), dim=-1)
      outputprobs3 = torch.nn.functional.softmax(self.output3(x), dim=-1)
      return outputprobs1, outputprobs2, outputprobs3
</code></pre>
<p>The softmax probabilities indicate a certain action my agent will perform, but the agent has different actions based on different states. Because I know which actions my agent should perform, I want to train the policy through supervised learning. I am planning to use torch.nn.CrossEntropyLoss(), as it is a multi-classification problem. Moreover, during each episode, the model chooses from the output probabilities multiple times.</p>
<p>For example, let’s say that there are 3 states my agent can be in, A, B, and C. In state A, the agent uses output1, in state B, the agent uses output2, and in state C, the agent uses output 3. As a result, one example through an episode might be:</p>
<ul>
<li>Agent starts in state A: chooses action 9</li>
<li>Now agent is in state C: chooses action 2</li>
<li>Now agent is in state B: chooses action 4</li>
<li>Now agent is in state A: chooses action 5</li>
</ul>
<p>Here are my following questions:</p>
<p>How would I train this policy through supervised learning? The website for CrossEntropyLoss says I need a input of shape (N, C). So my C here would be 10, 5, and 3 respectively. My thinking is that I would need 3 of these inputs, one for each output layer. As a result, would I need a separate loss function for each output layer, or can I just use one loss function for all of them? <strong>Would propagating the loss for one of the output layers affect the others negatively through supervised learning?</strong></p>
<p>Moreover, I would like more clarification on how to obtain the logits needed for the input to CrossEntropyLoss. In order to obtain this, would I use <code>torch.logit(outputprobs1)?</code></p>
<p>Any help with this is much appreciated, thank you!</p>

pascal_notsawo: <p>x that you pass to your forward function represents what?</p>
<p>As I mentioned <a href="https://discuss.pytorch.org/t/is-this-usage-of-mseloss-correct/120392/2">here</a>, one loss function is enough, it’s just one function.</p>
In Training, swapaxis and squeeze is OK for loss.backward()?

SungmanHong: <p>In my code, I have many swapaxis and squeeze in the nn class layer.<br>
for example,</p>
<pre><code class="lang-auto">    def forward(self, pixel, video_length):
        for avg_pooling, model in enumerate(self.batch):
            pixel = model(pixel)
            if avg_pooling == 1:
                avg_pooling_2d = nn.functional.adaptive_avg_pool2d(pixel, 1)
                avg_pooling_2d = torch.swapaxes(pixel, 2, 0)
                avg_pooling_2d = torch.swapaxes(avg_pooling_2d, 1, 2)
                avg_pooling_2d = torch.squeeze(pixel)
                avg_pooling_2d = torch.unsqueeze(avg_pooling_2d)
                input = self.ReLU(avg_pooling_2d)
                return input
</code></pre>
<p>My problem is, when loss.backward is applied, the backpropagation process is automatically recover the swapaxis and squeeze process which is originally located at the <code>def forward()</code> ?</p>
<p>In my opinion… the loss is gradually increasing when the training epoch is increasing… So I doubt that the swapaxis in the training session may broke the dimension order at the backpropagation process. Anyone knows that swapaxis and squeeze is OK for loss.backward in the PyTorch? Please help!!</p>
<hr>
<p>SIMPLE SUMMURY : Can I use swapaxis in forward()? (no problem in loss.backward() ?)</p>
<hr>
<p>Thanks.</p>
Disabling cudnn algorithms

saar-eliad: <p>Hi, is there a PyTorch equivalent for the following and alike?</p>
<p><a href="https://docs.nvidia.com/deeplearning/frameworks/tensorflow-user-guide/index.html#tf_enable_winograd_nonfused" class="onebox" target="_blank" rel="noopener nofollow ugc">https://docs.nvidia.com/deeplearning/frameworks/tensorflow-user-guide/index.html#tf_enable_winograd_nonfused</a></p>
<p>I wonder, in TF it is integrated into the framework,<br>
but maybe it is possible to do the same from the cudnn dynamic lib?</p>

ptrblck: <p>No, currently you cannot disable specific cudnn algorithms manually. We are working on the enablement of the cudnn v8 API, which allows more flexible filtering.</p>
Why do some inputs need an axis for batch size but some don't?

vishak_bharadwaj: <p>I’m building a simple network that takes in two numbers and learns how to add them.</p>
<pre><code class="lang-auto">import torch 

add1= torch.randint(0,9,size=[6000])
add2= torch.randint(0,9,size=[6000])
add_sum = add1 + add2
</code></pre>
<p>This a pretty simple network</p>
<pre><code class="lang-auto">from torch import nn
from torch.nn import functional as F 

class Net(nn.Module):

  def __init__(self):
    super().__init__()
    self.linear1 = nn.Linear(2,20)
    self.linear2 = nn.Linear(20,1)

  def forward(self,x1,x2):
    inp = torch.cat((x1[None],x2[None])).float()
    out = self.linear1(inp)
    out = F.relu(out)
    out = self.linear2(out)

    return out
</code></pre>
<p>Here’s the training loop</p>
<pre><code class="lang-auto">net = Net()
optim = torch.optim.AdamW(net.parameters(),lr=0.1)
criterion = nn.MSELoss()

for i in range(len(add1)):

  out = net(add1[i],add2[i])
  loss = criterion(out,add_sum[i].float())
  optim.zero_grad()
  loss.backward()
  optim.step()

  if i%500==0: print(loss)
</code></pre>
<ol>
<li>Now the input here doesn’t have batch size, yet it works. But sometimes pytorch inference doesn’t work without a batch size. Why is that?</li>
<li>If I’m trying to input a vector of n features to an NN that starts with a linear layer<br>
should be the shape (n,) or (n,1) or (1,n) ? Pretty confused about that here.</li>
<li>Is the way I’m handling the input the right way? Or is there a better way to do it?</li>
</ol>

vishak_bharadwaj: <p>Essentially a broader question would be, is there a guide on the shapes and types the tensors have to be for different models and loss functions?</p>

ptrblck: <p>Yes, the <a href="https://pytorch.org/docs/stable/index.html">docs</a> mention the expected shape for each layer and I would stick it to it. My general rule is that a batch dimension is expected in <code>nn.Module</code>s. While e.g. linear layers could work with an input having a single dimension, you would have to verify what’s applied internally (which dimension is broadcasted etc.), so I would prefer to use the documented approach.</p>
Memory overflow issue in CUDA when trying to make a prediction for a trained model

Viktor_Andriichuk: <p>Faced the following error:</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-39-94bd54aad11f&gt; in &lt;module&gt;()
      4 # get predictions for test data
      5 with torch.no_grad():
----&gt; 6   preds = model(test_seq.to(device), test_mask.to(device))
      7   preds = preds.detach().cpu().numpy()

8 frames
/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
   1914         # remove once script supports set_grad_enabled
   1915         _no_grad_embedding_renorm_(weight, input, max_norm, norm_type)
-&gt; 1916     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
   1917 
   1918 

RuntimeError: CUDA out of memory. Tried to allocate 11.22 GiB (GPU 0; 15.78 GiB total capacity; 12.10 GiB already allocated; 2.18 GiB free; 12.30 GiB reserved in total by PyTorch)
</code></pre>
<p>The prediction code is as follows:</p>
<pre><code class="lang-auto"># specify GPU
device = torch.device("cuda")

# get predictions for test data
with torch.no_grad():
  preds = model(test_seq.to(device), test_mask.to(device))
  preds = preds.detach().cpu().numpy()
</code></pre>
<p>What to do?</p>

ptrblck: <p>You could try to reduce the memory usage by reducing e.g. the batch size. I also assume you’ve made sure the GPU is completely empty before running the script and no other process is using it.</p>
Smpling binary map from a grid (generating occlusion-meaning mask from depth warping)

111536: <p>Hello, I implemented depth warping just using <a href="https://github.com/sunghoonim/DPSNet/blob/master/inverse_warp.py" rel="noopener nofollow ugc">inverse_warp.py</a> from DPSNet for 480*640 image and its depth map</p>
<p>I want to create a binary map with the target coordinates of depth warping as 1 and the other coordinates as 0. In other words, it means displaying the occlusion-exposed background as 0 and the non-occlusion as 1. (What I want is not an occlusion mask, but it represents occlusion by marking the region that was occluded as 0.)</p>
<pre><code class="lang-auto">    src_pixel_coords = cam2pixel(cam_coords, proj_cam_to_src_pixel[:,:,:3], proj_cam_to_src_pixel[:,:,-1:], padding_mode)  # [B,H,W,2]
    projected_feat = torch.nn.functional.grid_sample(feat, src_pixel_coords, padding_mode=padding_mode)

    return projected_feat, src_pixel_coords
</code></pre>
<p>To achieve this, I used invers_warp.py to return src_pixel_coords (i.e. target coordinates), which is passed as an argument to the torch.nn.functional.grid_sample() function.</p>
<p>From this, src_pixel_coords value, I can produce the binary map I desire by following numpy code:</p>
<pre><code class="lang-auto">warped_img, src_pixel_coords = inverse_warp( ... )

img_height = 480
img_width = 640

coords = (src_pixel_coords + 1) / 2       #0-1 normalize
x_coords = coords[0,:,:,0] * img_width  #batch_size = 1
y_coords = coords[0,:,:,1] * img_height

x_coords = x_coords.floor().clamp(0,img_width-1).cpu().numpy()
y_coords = y_coords.floor().clamp(0,img_height-1).cpu().numpy()

mask = torch.zeros(img_height, img_width)
for y in range(img_height):
    for x in range(img_width):
        y_ = np.clip(2*y-int(y_coords[y,x]), 0, img_height-1)
        x_ = np.clip(2*x-int(x_coords[y,x]), 0, img_width-1)
        mask[y_, x_] = 1
</code></pre>
<p>And the output:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/b/e/bed78e027d375610d3f8d106e4daff8ab8dcfb04.png" alt="6_back_mask" data-base62-sha1="regxGgOMBacvSVB6bnMIEpCAsqE" width="640" height="480"></p>
<p>It works properly and produce binary map well. However, there are several problems with this code.</p>
<ol>
<li>It is implemented in numpy, therefore it can’t be computed on GPU.</li>
<li>Of course, it doesn’t work when batch size&gt; 1.</li>
<li>Bad time complexity.</li>
</ol>
<p>I would like to implement this numpy code as functions for a pytorch tensor and solve the above three problems. However, I am not sure how to solve this problem. Can you help?</p>
Running multiple batches trough model before backpropagation

huberl: <p>In my code I’m trying to implement the semi-supervised learning method FixMatch. However, I’m not sure if I implemented it correctly. Can I first run the labeled data and then the unlabeled data trough the model, or will this ruin my gradient computation? In <a href="https://github.com/kekmodel/FixMatch-pytorch/blob/ced359021e0b48bf3be48504f98ea483579c6e71/train.py#L349-L350" rel="noopener nofollow ugc">another implementation</a> I saw that they combine both to yield a single input.</p>
<p>In my code, <code>input_la</code> is a batch of labeled samples, <code>input_ul_weak</code> a batch of weakly augmented samples and  <code>input_ul_strong</code> a batch of strongly augmented samples.</p>
<pre><code class="lang-auto"># Get model predictions on labeled data
  out_la = model(input_la)

  # We don't need the gradient here since we are only using the pseudo labels
  with torch.no_grad():
    out_ul_weak = model(input_ul_weak)
  
  out_ul_strong = model(input_ul_strong)

  # Get pseudo label from weakly augmented sample
  conf, pseudo_label = F.softmax(out_ul_weak, dim=1).max(axis=1)
  
  # Only use confident pseudo labels
  mask = conf &gt; threshold

  # Losses
  loss_la = F.cross_entropy(out_la, label_la)
  loss_ul = F.cross_entropy(out_ul_strong[mask], pseudo_label[mask])

  loss_combined = loss_la + lambda_u * loss_ul

  model.zero_grad()
  loss_combined.backward()
  optimizer.step()
</code></pre>

ptrblck: <p>Your can execute multiple forward passes, calculate different losses, combine them, and calculate the gradients using the accumulated loss, so your code looks alright.<br>
Are you seeing any unexpected behavior using this approach?</p>

huberl: <p>I get a somewhat weird behavior. If I train the code without the unlabeled data</p>
<pre><code class="lang-auto">out_la = model(input_la)
loss_la = F.cross_entropy(out_la, label_la)

model.zero_grad()
loss_la.backward()
optimizer.step()
</code></pre>
<p>it converges pretty fast. If I use my original code instead and set the threshold to 1, it obviousely uses no unlabeled data since the mask is False everywhere. What surprises me is that even in this case it does not really converge and acts totally differently than the purely supervised code.</p>
<p>In <a href="https://discuss.pytorch.org/t/masking-input-to-loss-function/121830">my other question</a> I tried finding out if its a masking issue, but I’m not sure.</p>
Why do I get a segmentation fault for memory checking?

lingvisa: <p>I am using Python 1.7</p>
<pre><code class="lang-auto">In [1]: import torch
   ...: from torchvision.models import vgg19
   ...: 
   ...: device = torch.device("cuda:0")

In [2]: 

In [2]: memory = torch.cuda.memory_allocated(device)
Segmentation fault (core dumped)
</code></pre>
<p>And my GPU info:</p>
<pre><code class="lang-auto">~# nvidia-smi
Fri May 21 13:13:27 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.87.01    Driver Version: 418.87.01    CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla V100-SXM2...  On   | 00000000:00:10.0 Off |                    0 |
| N/A   33C    P0    56W / 300W |  15323MiB / 16130MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla V100-SXM2...  On   | 00000000:00:11.0 Off |                    0 |
| N/A   33C    P0    57W / 300W |    462MiB / 16130MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  Tesla V100-SXM2...  On   | 00000000:00:12.0 Off |                    0 |
| N/A   39C    P0    56W / 300W |    464MiB / 16130MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  Tesla V100-SXM2...  On   | 00000000:00:13.0 Off |                    0 |
| N/A   55C    P0   283W / 300W |  15965MiB / 16130MiB |     99%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+
</code></pre>
<p>However, if I check on terminal, it has no segmentation fault:</p>
<pre><code class="lang-auto"># python -c 'import torch; print(torch.cuda.memory_allocated(torch.cuda.device("cuda:0")))'
0
</code></pre>
<p>Also, the GPU is used fully, but why this check also returns 0?</p>
<pre><code class="lang-auto"># python -c 'import torch; print(torch.cuda.memory_allocated(torch.cuda.device("cuda:3")))'
0
</code></pre>

xshaun: <p>The code you provide looks fine; I run it just now, and there is no error occurred. If possible, you can launch a new terminal to try again or provide more detailed information about your exp env, like torch version.</p>
Save Pruned Model After Pruning

leimao: <p>Pruning could also be used as a sort of neural architecture search method.<br>
Say, I have a two-layer convolutional neural network.</p>
<pre><code class="lang-auto">conv1 = torch.nn.Conv2d(in_channels=3, out_channels=12)
conv2 = torch.nn.Conv2d(in_channels=12, out_channels=8)
</code></pre>
<p>If I used structured pruning and say the last three channels of the conv2 kernel was pruned to 0. Then the useful architecture of the pruned model should be</p>
<pre><code class="lang-auto">conv1 = torch.nn.Conv2d(in_channels=3, out_channels=9)
conv2 = torch.nn.Conv2d(in_channels=9, out_channels=8)
</code></pre>
<p>This architecture with the remaining parameters could be saved as a new model. The size of this new model should be smaller and the inference speed of the new model should be faster than the unpruned one.</p>
<p>I wonder if PyTorch is planning to add this interface or not.</p>

Ankita_Makker: <p>any updates on this?</p>

grid_world: <p>If there aren’t any work on this, maybe we can create an extended class and create a push request?</p>

grid_world: <p><a class="mention" href="/u/leimao">@leimao</a> Shouldn’t the ‘useful architecture’ <em>post structured pruning</em> be as:</p>
<pre><code class="lang-auto">conv1 = torch.nn.Conv2d(in_channels = 3, out_channels = 9)
conv2 = torch.nn.Conv2d(in_channels = 9, out_channels = 5)
</code></pre>
<p>because <strong>you are only pruning the last three channels/filters/kernel maps of conv2</strong> layer? conv1 layer is untouched/unpruned.</p>
Unable to export model to ONNX model

111532: <p>When I tried to export my trained pytorch model to ONNX format, I encounter the error: <code>Cannot insert a Tensor that requires grad as a constant. Consider making it a parameter or input, or detaching the gradient</code></p>
<p>After searching on board, I found multiple cases that results in same error, but I didn’t find a solution suitable for my case.</p>
<p>Here is my code:</p>
<pre><code class="lang-auto"># Define a function: input raw data =&gt; preprocessing &amp; model inference =&gt; prediction
from PIL import Image
# Preprocess image for PyTorch data architecture
from torchvision import transforms

data_preprocess = transforms.Compose([
      transforms.ToTensor(),
      transforms.Normalize(0.456, 0.225)
    ])

device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
def process_for_model(img):
  '''
  Process image array for PyTorch model
  img: input tensor of an image
  return: Tensor image prepared to input to the model
  '''
  #img = Image.fromarray(img).convert('RGB')
  img = data_preprocess(img) # remember to do data preprocessing as in the training stage!! This strongly influence the testing performance
  img = img.to(device)
  img = img.view(1, 3, 889, 929)

  return img

def pipeline(input_batch, preprocess, inference):
  processed_data = preprocess(input_batch)
  output = inference(processed_data)
  return output

def final_pipeline(input_batch):
  prediction = pipeline(input_batch)
  return prediction

class mypipeline(torch.nn.Module):
  def __init__(self):
    super().__init__()

  def forward(self, input_batch):
    input_batch = input_batch.cpu().detach().numpy().astype(np.float32)
    processed_data = process_for_model(input_batch)
    
    print("processed data:", processed_data)
    self.output = pre_model(processed_data).cpu().detach()
  
    return self.output



input_shape = (889, 929, 3)
dummy_input = torch.randint(0, 255, size = (889, 929, 3), device = torch.device("cuda:0"))
pre_model.eval()
pipeline = mypipeline()
pipeline.eval()

with torch.no_grad():
  torch.onnx.export(pipeline,
         dummy_input,
         "txt_overlap_pip.onnx",
         verbose = False,        
        )
</code></pre>
<p>I’ve numpy array in the workflow, which I’m not sure if it would be the cause of problem.<br>
Any advice is appreciated, thank you!</p>
How to speed up Non-Zero op?

seunghyun_lee: <p>Hi, my CNN is quite simple 6 Conv-BN-Relu stacked network and it should run on CPU.</p>
<p>The main speed bottle neck is in “nonzero” op in the last.<br>
It takes almost 90% of the whole processing time, so I need to optimize this op more.</p>
<p>I found torch uses OpenMP or TBB’s “parallel for” and Intel and facebook collaborated to make a faster one called IPEX-pytorch(mabye it is out of box).</p>
<p>Does anybody can tell me how to implement a faster NonZero op?</p>
<p>I tried Openvino and Onnxruntime.</p>
<ol>
<li>Openvino can optimize some ops with weight but it doesn’t support Nonzero op.</li>
<li>Onnxruntime is almost same as torch’s nonzero op.</li>
</ol>
<p>Thank you!</p>
Swapaxis can be used for backpropagation?

SungmanHong: <p>I made torch.nn.module layer<br>
which modifies the axis of input tensors with swapaxis.</p>
<p>What I want to know is, when the backpropagation of loss is performed,<br>
"the axis of loss is also automatically swapped? "<br>
I wonder, when I modify some axis of input tensor, the loss is somewhat increased and not reducing, So I want to check this issue. Thanks.</p>
Passing tensors between GPUs just before loss calculation

ClaartjeBarkhof: <p>Hi there,</p>
<p>I am currently using Distributed Data Parallel to achieve multi GPU training. So far, I did not need to send data across GPUs, because I could make use of the fact that in the backward pass the gradients are gathered from all GPUs before updating the models on the different GPUs automatically by the Distributed Data Parallel class.</p>
<p>However, now, I would like to extend / change my loss function with a calculation that requires the data on all GPUs. So, I would like to manually / explicitly exchange tensors between GPUs before calling backward. Let’s say the loss is calculated from two terms:</p>
<ol>
<li>One term, like a reconstruction loss term, that can be calculated on all GPUs separately and may be treated as normal, or additive in terms of gradients.</li>
<li>A second term, like a statistic on all samples in batches across different GPUs, for which I would like to have the GPUs sync / exchange data, before doing a loss calculation and backward pass.</li>
</ol>
<p>So, I need to know A) how to achieve this data passing for the second term, but B) also how to compute the loss and perform a backward pass with these two types of losses together.</p>
<p>Any help is much appreciated!</p>
<p>Thanks,</p>
<p>Claartje</p>

ptrblck: <p>You could take a look at the <a href="https://pytorch.org/tutorials/intermediate/dist_tuto.html#collective-communication">Collective Communication docs</a> which explain how tensors can be scattered, gathered, and reduced in a distributed setup.<br>
Once you’ve made sure the data is available on the desired devices, I assume you can calculate the loss and let DDP compute the gradients etc.</p>

wanchaol: <p>agreed with <a class="mention" href="/u/ptrblck">@ptrblck</a>. <a class="mention" href="/u/claartjebarkhof">@ClaartjeBarkhof</a>  You can manually send the data across gpus with the communication ops provide by the distributed package, in this case you might consider using <a href="https://pytorch.org/docs/stable/distributed.html#torch.distributed.all_reduce" rel="noopener nofollow ugc">allreduce</a>, scatter or <a href="https://pytorch.org/docs/stable/distributed.html#torch.distributed.reduce_scatter" rel="noopener nofollow ugc">reduce_scatter</a>. If you also want to automatically support backward for those data communications, consider using the <a href="https://github.com/pytorch/pytorch/blob/5211eeb22b17f706462bff4815f3c4bbf1af1c0c/torch/distributed/nn/functional.py#L6" rel="noopener nofollow ugc"><code>torch.distributed.nn</code></a> to do those communications</p>
How to select specific channels in an image tensor?

meshghi: <p>If the input has a shape [N,C,H,W], the output’s shape should be [N,masked_C,H,W]. Where masked_C is the channels of interest. For example, if C=4, then, masked_C is indices 0,2,3</p>

InnovArul: <p>Can you try this?</p>
<pre><code class="lang-auto">tensor=tensor[:, [0,2,3]]

</code></pre>
Colab not working

Ray1: <p>I am trying to run <a href="https://colab.research.google.com/drive/1Aa0eKSmyYef1gORvlHv7EeQzSVRb30eL?usp=sharing" class="inline-onebox" rel="noopener nofollow ugc">Google Colaboratory</a> but the program crashes in the second cell. How can I fix the error?</p>

pascal_notsawo: <p>Briefly:</p>
<ul>
<li>you have installed :</li>
</ul>
<pre><code class="lang-auto">!pip install -q torch-scatter -f https://pytorch-geometric.com/whl/torch-1.7.0+cu101.html
!pip install -q torch-sparse -f https://pytorch-geometric.com/whl/torch-1.7.0+cu101.html
!pip install -q torch-geometric
!pip install ogb
</code></pre>
<ul>
<li>when you do this import :</li>
</ul>
<pre><code class="lang-auto">from torch_geometric.datasets import TUDataset

</code></pre>
<ul>
<li>you get this error :</li>
</ul>
<pre><code class="lang-auto">---------------------------------------------------------------------------
/usr/local/lib/python3.7/dist-packages/torch_geometric/__init__.py in &lt;module&gt;()
      3 
      4 from .debug import is_debug_enabled, debug, set_debug
----&gt; 5 import torch_geometric.data
      6 import torch_geometric.transforms
      7 import torch_geometric.utils

/usr/local/lib/python3.7/dist-packages/torch_geometric/data/__init__.py in &lt;module&gt;()
----&gt; 1 from .data import Data
      2 from .temporal import TemporalData
      3 from .batch import Batch
      4 from .dataset import Dataset
      5 from .in_memory_dataset import InMemoryDataset

/usr/local/lib/python3.7/dist-packages/torch_geometric/data/data.py in &lt;module&gt;()
      6 import torch
      7 import torch_geometric
----&gt; 8 from torch_sparse import coalesce, SparseTensor
      9 from torch_geometric.utils import (contains_isolated_nodes,
     10                                    contains_self_loops, is_undirected)

/usr/local/lib/python3.7/dist-packages/torch_sparse/__init__.py in &lt;module&gt;()
     13 ]:
     14     torch.ops.load_library(importlib.machinery.PathFinder().find_spec(
---&gt; 15         f'{library}_{suffix}', [osp.dirname(__file__)]).origin)
     16 
     17 if torch.cuda.is_available():  # pragma: no cover

/usr/local/lib/python3.7/dist-packages/torch/_ops.py in load_library(self, path)
    102             # static (global) initialization code in order to register custom
    103             # operators with the JIT.
--&gt; 104             ctypes.CDLL(path)
    105         self.loaded_libraries.add(path)
    106 

/usr/lib/python3.7/ctypes/__init__.py in __init__(self, name, mode, handle, use_errno, use_last_error)
    362 
    363         if handle is None:
--&gt; 364             self._handle = _dlopen(self._name, mode)
    365         else:
    366             self._handle = handle

OSError: /usr/local/lib/python3.7/dist-packages/torch_sparse/_convert_cuda.so: undefined symbol: _ZN6caffe28TypeMeta21_typeMetaDataInstanceIdEEPKNS_6detail12TypeMetaDataEv

</code></pre>
Does pytorch.multiprocessing require more GPU memory?

ljj7975: <p>my training code takes up about 9GB and I have 2080Ti which has 11GB of memory</p>
<p>when I am running this on a single GPU,<br>
the process is pretty stable, as the GPU usage stays under 10GB</p>
<p>However, when I add extra 2080Ti device and use torch.multiprocessing for parallel training,<br>
I have noticed that it takes up little more GPU and rarely cause out of memory issue.</p>
<p>I am wondering if the increase in GPU memory usage is expected.<br>
The documentation says that when sharing CUDA memory, the original tensor will exist until it’s moved to the other one (<a href="https://pytorch.org/docs/stable/multiprocessing.html#sharing-cuda-tensors" class="inline-onebox" rel="noopener nofollow ugc">Multiprocessing package - torch.multiprocessing — PyTorch 1.8.1 documentation</a>).<br>
Is this why multiprocessing case consumes little more memory?</p>
Change output shape of any CNN architecture

sammlapp: <p>Is there a general way to change the output shape of any Pytorch architecture?</p>
<p>Based on this article (<a href="https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Finetuning Torchvision Models — PyTorch Tutorials 1.2.0 documentation</a>) it seems like each architecture requires a different method to change the number of output nodes (classes) - for instance,<br>
for resnet18 you can replace model.fc with a nn.Linear layer but for vgg11_bn you have to replace the model.classifier[6] layer. Unfortunately, that article does not have all of the currently implemented PyTorch architectures. In general as more architectures are added it would be hard to keep adding new wrapper functions for changing the output shape. It would be very helpful if there were a consistent way to modify the output shape of any pytorch architecture.</p>
Explicit gradient calculation over two models

Jack_Han: <p>Hi,</p>
<p>I have one problem for the gradient calculation in pytorch:<br>
I have two models which have following calculate:<br>
y1 = model1(w1,x)<br>
Here y1 is a vector, w1 is a matrix,<br>
y2 = model2(w2,y1)<br>
Here y2 is a scalar<br>
I have obtained the gradient of following in pytorch:<br>
dy2/dy1 which is a vector, and dy1/dw1 which is a matrix,</p>
<p>I must use this two gradients explicitly to complete the chain rule,<br>
how can I  calculate the gradient of dy2/dw1 in pytorch?</p>
<p>Thanks</p>
Pre-padding a sequence

samip_timalsena: <p>Hi, Is there a way of pre-padding a text sequence. The <strong>pad_sequence</strong> implementation gives post-padding result.</p>
<pre><code class="lang-auto">train_data = pad_sequence(train_data, batch_first=True, padding_value=vocab.stoi['&lt;pad&gt;'])
train_data[0]
out: tensor([60, 15,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1, 1,  1,  1,  1])
</code></pre>
<p>The result i want to get is:</p>
<pre><code class="lang-auto">out: tensor([1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1, 1,  1,  1,  1, 60, 15])
</code></pre>