akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Inference speed drops off dramatically after first few batches

Duncan_Freeman: <p>Hey y’all, I’m using <a href="https://github.com/mit-han-lab/pvcnn" rel="noopener nofollow ugc">PVCNN</a> with custom weights to do semantic segmentation on point clouds. For the first 6 batches or so, inference occurs quickly - but thereafter the speed is reduced considerably. I did not trust measurements via <code>time()</code> from python, so I checked the profiler and sure enough the result is consistent with what I saw: <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/9/f9540d48b31f5ade530a01fe444da1d82d40d016.png" data-download-href="https://discuss.pytorch.org/uploads/default/f9540d48b31f5ade530a01fe444da1d82d40d016" title="2021-01-04-132643_1622x231_scrot"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/9/f9540d48b31f5ade530a01fe444da1d82d40d016_2_690x98.png" alt="2021-01-04-132643_1622x231_scrot" data-base62-sha1="zzEYaybh5ndB5I79800cJIqrs8K" width="690" height="98" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/9/f9540d48b31f5ade530a01fe444da1d82d40d016_2_690x98.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/9/f9540d48b31f5ade530a01fe444da1d82d40d016_2_1035x147.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/9/f9540d48b31f5ade530a01fe444da1d82d40d016_2_1380x196.png 2x" data-dominant-color="DAE2EC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">2021-01-04-132643_1622x231_scrot</span><span class="informations">1622×231 22 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Frustratingly, enabling <code>use_cuda=True</code> in the profiler makes it such that all batches infer at the same, slow pace; total inference time increases by seconds when CUDA profiling is enabled. My inference loop looks like so:</p>
<pre data-code-wrap="python"><code class="lang-python">points = torch.from_numpy(my_pointclouds)
loader = DataLoader(points, batch_size=40, num_workers=0, pin_memory=True)
with torch.no_grad() and profiler.profile(record_shapes=True) as prof:
    for batch in loader:
        outputs = model(batch.to(configs.device, non_blocking=True))
        del batch, outputs
prof.export_chrome_trace("trace.json")
</code></pre>
<p>The dataset should not be too large for my 8 GB GTX 1080, as it only takes up a few MB.<br>
What could possibly be negatively affecting the inference speed of PVCNN? Why does profiling CUDA have such an impact? Thanks</p>

ptrblck: <p>The “non-CUDA” profiling might not synchronize the code, so your times and the profile might just show the kernel launch times and dispatching, while <code>use_cuda=True</code> would show the actual CUDA runtime.</p>

Duncan_Freeman: <p>I thought so too, but setting <code>use_cuda=True</code> actually measurably changes the running time of the script as a whole</p>
Equivalent of ignore_index in a continuous valued loss

catosphere: <p>Hello everyone,</p>
<p>I have continuous valued data of variable lengths for training a convolutional auto-encoder.<br>
Something similar to having e.g. images of variable width.</p>
<p>I read that NLL loss has an ignore_index which allows part of the target not to contribute into the loss gradient. In the case of categorical data, this would allow padding all data to the maximum length (e.g. 300 points) with the ignore_index value in order to ommit the padding in the loss.</p>
<p>Would there be any efficient way to implement something similar for continuous targets please (e.g. using MSE loss) ?<br>
rather than a for loop over each batch element to crop and compute the loss individually<br>
it could be defining a target continuous value to ignore, or something less numerically sensitive like a floor value under which all values are ignored (e.g. my data is in [-1,1] and I could set ignore value to &lt; -1 and pad with -1.5)</p>
<p>I imagine I could bin the values (e.g. mu-law) but before that I am interested of any way not to go through binning</p>
<p>Second question, let’s say I want to train a VAE which will also have a regularization of the encoding.<br>
Because the encoder is passed over the padded data (as the input), that latent loss will be computed taking into account the padding. Is there any way to prevent some input pixels not to contribute to any gradients (neither the regularization nor the reconstruction loss).</p>
<p>I am curious of your suggestions ! Right now I feel it’s either not really feasible, or only applicable to training with a batch_size = 1 and some latent pooling that is invariant to input size …</p>
<p>Thanks !</p>
The difference of `torch.nn.quantized.FloatFunctional.add` and `torch.nn.quantized.FloatFunctional.add_scalar`?

feiyuhuahuo: <p>What’s the difference between  <code>torch.nn.quantized.FloatFunctional.add</code> and <code>torch.nn.quantized.FloatFunctional.add_scalar</code>? Is the former one used for <code>tensor</code> + <code>tensor</code> and the latter one used for <code>tensor</code> + <code>scalar</code> ? Or anything other?</p>

Vasiliy_Kuznetsov: <aside class="quote no-group" data-username="feiyuhuahuo" data-post="1" data-topic="108878">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/feiyuhuahuo/48/15165_2.png" class="avatar"> feiyuhuahuo:</div>
<blockquote>
<p>Is the former one used for <code>tensor</code> + <code>tensor</code> and the latter one used for <code>tensor</code> + <code>scalar</code> ? Or anything other?</p>
</blockquote>
</aside>
<p>Yes, this is correct.  You can check this out in the source code, here: <a href="https://github.com/pytorch/pytorch/blob/b89827b73f7881a8108518452768654894862e5d/torch/nn/quantized/modules/functional_modules.py#L42" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/b89827b73f7881a8108518452768654894862e5d/torch/nn/quantized/modules/functional_modules.py#L42</a></p>
NL dimension in Kornia's ScalePyramid

sunder: <p>I want to use a scale pyramid to extract resize invariant features for an image similarity project. I’m using resnet18 trained with triplet loss. I want to extract features at multiple scales and average pool them.</p>
<p>But when I pass the image tensor through kornia.geometry.transform ScalePyramid, it is returning images with shape B,C,NL,H,W.</p>
<p>I’m not sure how to understand the NL dimension. I need tensors in the shape B,C,H,W. How can I drop this NL dimension?</p>
<p>I also want to train a network that is invariant to scale. Can I use this scale pyramid during training? If yes, how to accommodate the NL dimension.</p>

ptrblck: <p>I assume <code>NL</code> is the number of scale levels, but cannot find the exact definition in the docs.<br>
CC <a class="mention" href="/u/edgarriba">@edgarriba</a>, who would know it <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ducha-aiki: <p>NL is number of scale levels of the scale space, see SIFT paper, Figure 1 on the left. This mean to consequently blur the image more and more, which for linear filters is equivalent to resizing.</p>
<p>If you just need a pyramid, which contains HW, H/2 W/2, H/4 W/4 and so on - use <a href="https://kornia.readthedocs.io/en/latest/geometry.transform.html?highlight=build_pyramid#kornia.geometry.transform.build_pyramid" class="inline-onebox" rel="noopener nofollow ugc">kornia.geometry.transform — Kornia documentation</a> function instead. It does not have levels unlike scale-space pyramid, that is a mistake in documentation (fixed in master)</p>
<aside class="onebox pdf">
  <header class="source">
      <a href="https://people.eecs.berkeley.edu/~malik/cs294/lowe-ijcv04.pdf" target="_blank" rel="noopener nofollow ugc">people.eecs.berkeley.edu</a>
  </header>
  <article class="onebox-body">
    <a href="https://people.eecs.berkeley.edu/~malik/cs294/lowe-ijcv04.pdf" target="_blank" rel="noopener nofollow ugc"><span class="pdf-onebox-logo"></span></a>
<h3><a href="https://people.eecs.berkeley.edu/~malik/cs294/lowe-ijcv04.pdf" target="_blank" rel="noopener nofollow ugc">lowe-ijcv04.pdf</a></h3>

<p class="filesize">501.00 KB</p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Would you please tell me how is the `torch.argsort` implemented?

coincheung: <p>Hi,</p>
<p>I am learning the source code of pytorch, and I would like to know how can pytorch compute the rank along cerain dimension of a tensor. However, the code is too abstract to understand, so I am asking what method is used to compute this please ? Did you use thrust combined with a for loop to compute it many times, or you wrote a cuda kernel with merge sort or methods like this?</p>

albanD: <p>Hi,</p>
<p>argssort is implemented by doing <code>sort(same_args)[1]</code> <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> You can see that implementation here: <a href="https://github.com/pytorch/pytorch/blob/22a34bcf4e5eaa348f0117c414c3dd760ec64b13/aten/src/ATen/native/LegacyDefinitions.cpp#L27-L29" class="inline-onebox">pytorch/LegacyDefinitions.cpp at 22a34bcf4e5eaa348f0117c414c3dd760ec64b13 · pytorch/pytorch · GitHub</a></p>
<p>And for sort, you can see here that it comes from the old THC library: <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/native_functions.yaml#L6340" class="inline-onebox">pytorch/native_functions.yaml at master · pytorch/pytorch · GitHub</a><br>
And you can find the THC implementation in a couple files there: <a href="https://github.com/pytorch/pytorch/blob/070a30b265c1f4f404081ffefa7c41383750cc2e/aten/src/THC/generic/THCTensorSort.cu" class="inline-onebox">pytorch/THCTensorSort.cu at 070a30b265c1f4f404081ffefa7c41383750cc2e · pytorch/pytorch · GitHub</a></p>
When to call torch,cuda.synchronize()?

mipt98: <p>Dear All,</p>
<p>I have a strange problem with a code snippet which looks like the one below</p>
<pre><code class="lang-auto"># model doesn't contain any trainable parameters
def comp(model, x):
    x.requires_grad_()
    y = model(x)
    y.backward()
    g = x.grad
    return x.detach_(), y.detach_(), g

def run(model, x):
    x, y, g = comp(model, x)

    while True:
        some plain compute with x, y, and g, no backward() calls
        torch.cuda.synchronize()
        call some function that computes svd
        x.copy_(some z)
        x.grad.zero_()
        x, y, g = comp(model, x)
</code></pre>
<p>The code runs correctly on the cpu; however, in order to get the correct result on the gpu I have to insert the torch.cuda.synchronize() call. It also works correctly on the gpu if the sync is replaced by a python sleep call. Could you suggest what problem it may indicate?<br>
Thanks!</p>

albanD: <p>Hi,</p>
<p>If you only use pytorch’s API, you should never need to call synchronize yourself.<br>
Can you share a small code sample (that we can run on colab) that shows the bad result?</p>
Using trained model for prediction with variable input batch size

Sascha: <p>Hi</p>
<p>So i was wondering:<br>
I’ve trained my model with a batch size of, lets say 128.<br>
Now when i’m using my model in an application i might not always get 128 data points each time i want to predict something.<br>
Is there a possibility let the model predict only on the number of data points I’m feeding it, even though I’ve trained it for larger sizes?<br>
My first thought would be to use a padding method and just pad 0 tensors.<br>
And I’m guessing the same question would apply the other way around:<br>
Is there an integrated method to split larger inputs into batches of 128 and then handle the last batch, that probably isn’t 128?<br>
But is there an integrated method in Pytorch, or would i have to write it myself?</p>

user_123454321: <p>I am not sure why you would need 128 batch size input at prediction. Much of the time one would need to do prediction for a single sample (batch size of 1). Yes, some operations depend on the batch size like the BatchNorm, but during prediction, pytorch uses the running averages (computed during training) for the parameters for evaluation.</p>
Using Pytorch in AWS lambda Error- “Unable to import module 'lambda_function': No module named 'torch._C'”,

shivkhaira: <p>I am trying to use pytorch in AWS lambda but I keep getting this error. I am using<br>
<code>arn:aws:lambda:us-east-1:934676248949:layer:pytorchv1-py36:2</code><br>
as the layer for pytorch and this is my handler function</p>
<pre><code class="lang-auto">import unzip_requirements
import torch



def lambda_handler(event, context):
    
    return "Hello"
</code></pre>
<p>As in the function I am just only importing torch and it is giving me this error.</p>
<pre><code class="lang-auto">{
  "errorMessage": "Unable to import module 'lambda_function': No module named 'torch._C'",
  "errorType": "Runtime.ImportModuleError"
}
</code></pre>
Use pre trained resnet101 for regression data

anotherone_one: <p>Hello everyone!<br>
I wanted to use the resnet101 for a regression like problem. So, the input of the network is a image with an associated target (a number), and I want to get an output by training a model like regression.</p>
<p>What I am doing is adding a linear layer in the end of the resnet101 so the output if a single value.</p>
<p>model = models.resnet101(pretrained=True)</p>
<p>num_ftrs = model.fc.in_features</p>
<p>model.fc = nn.Linear(num_ftrs, 1)</p>
<p>Does this make sense? How would you suggest to use resnet or other pre trained cnn for continuous data?</p>

DAJINZI01: <p>I have the same problem about how to use a pre-trained model to my own work.</p>

anotherone_one: <p>my main question is basically how to use the pre trained resnet for continous data instead of categorical data! For categorical data you do what I put above and the linear layer is (num_ftrs, number of classes). Then, when extracting the prediction in the test phase you can use a softmax to get the probability of each class!</p>

VirginieBfd: <p>This solution looks correct.</p>
<pre><code class="lang-auto">model = models.resnet101(pretrained=True)
num_ftrs = model.fc.in_features
model.fc = nn.Linear(num_ftrs, 1)
</code></pre>
<p>Ensure you change you use a loss made for regression problems such as <code>torch.nn.MSELoss</code>.</p>
<p>You can use a network pre-trained on a classification problem and transfer it to a regression problem. You might need to unfreeze the last blocks to adapt it to your application.</p>
<p>The first blocks from ResNet or another net, will detect features such as edges, forms, which is mostly invariant when changing datasets.</p>
<p>Hope this helps!</p>

anotherone_one: <p>Thank you !! How can I unfreeze the last blacks ?</p>

VirginieBfd: <p>Something like:</p>
<pre><code class="lang-auto">for param in model.parameters():
     param.requires_grad = False # False when you freeze the layer / True when you want to train it
</code></pre>

anotherone_one: <p>Thank you! And other thing, why I need to unfreeze the last blocks? I think I am missing something here, like, theoretically I don’t understand very well why it is done like this.</p>

VirginieBfd: <p>This tutorial gives both the explanations and the example with ResNet:<br>
<a href="https://towardsdatascience.com/transfer-learning-picking-the-right-pre-trained-model-for-your-problem-bac69b488d16" class="onebox" target="_blank" rel="noopener nofollow ugc">https://towardsdatascience.com/transfer-learning-picking-the-right-pre-trained-model-for-your-problem-bac69b488d16</a></p>

anotherone_one: <p>Thank you so much for the help!! <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=9" title=":blush:" class="emoji" alt=":blush:"></p>
RuntimeError: element 0 of tensors does not require grad when trying to recreate TensorFlow function with torch.autograd.grad

ProGamerGov: <p>So, I am trying to recreate this TensorFlow function in PyTorch, but it doesn’t seem to be working:</p>
<pre><code class="lang-auto">
d_previous = tf.placeholder("float32")
d_logit = fwd_gradients(T("softmax2_pre_activation"), T(layer_name), d_previous)[0]


def fwd_gradients(ys, xs, d_xs):
  
  """Forward-mode pushforward analogous to the pullback defined by tf.gradients.
  With tf.gradients, grad_ys is the vector being pulled back, and here d_xs is
  the vector being pushed forward.
  
  By mattjj@google.com from
  https://github.com/renmengye/tensorflow-forward-ad/issues/2
  """

  v = tf.zeros_like(ys)
  g = tf.gradients(ys, xs, grad_ys=v)
  return tf.gradients(g, v, grad_ys=d_xs)
</code></pre>
<p>And my attempt at recreating the function in PyTorch:</p>
<pre><code class="lang-auto">def fwd_gradients(ys, xs):
    v = torch.zeros_like(ys)
    g = torch.autograd.grad(
            outputs=[ys],
            inputs=xs,
            grad_outputs=[v],
            retain_graph=True,
            
        )[0]
    out = torch.autograd.grad(
            outputs=[g],
            inputs=v,
            grad_outputs=torch.zeros_like(xs),
            retain_graph=True,
        )[0]
</code></pre>
<p>Currently the above PyTorch code results in the following error:</p>
<pre><code class="lang-auto">RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
</code></pre>
<p>How do I get the function working correctly?</p>

albanD: <p>Hi,</p>
<p>You need to give <code>create_graph=True</code> to the first call to grad if you plan on doing a backprop through it.</p>
<p>Also I don’t think you want to give <code>torch.zeros_like(xs)</code> for the second grad. It should be the <code>d_xs</code> that is in the tensorflow version</p>

ProGamerGov: <p>Thank you for the help! I guess I don’t need retain_graph to be true as well?</p>
<p>The function seems to work now!</p>
<pre><code class="lang-auto">def fwd_gradients(ys, xs, d_xs):
    v = torch.zeros_like(ys)
    v.requires_grad = True
    g = torch.autograd.grad(
            outputs=[ys],
            inputs=xs,
            grad_outputs=[v],
            create_graph=True,        
        )[0]

    out = torch.autograd.grad(
            outputs=[g],
            inputs=v,
            grad_outputs=d_xs,
        )[0]
</code></pre>
<p>Also, as PyTorch doesn’t use placeholders <code>d_xs</code> can probably be changed so that it’s just a ones tensor with the correct size.</p>

albanD: <p>If your function has a single input, you can give it a single one indeed to get the full gradients.<br>
Otherwise, you want to give the grad input (d_xs) that match what you want to compute.</p>
Best Siamese Net in PyTorch

Deeply: <p>I am looking for Siamese Net Open source to try it out for one of my projects.</p>
<p>I googled the topic and came across <a href="https://github.com/adambielski/siamese-triplet" rel="noopener nofollow ugc">siamese-triplet</a>, coded in 2017 based on PyTorch 0.4.</p>
<p>Would anyone suggest an alternative Siamese model?</p>
Sharing a parameter between multiple loss functions/graphs

algol: <p>I am trying to train several networks on similar datasets, and there are several constraints on the outputs that are common, even if the actual network outputs will be different. However, I don’t necessarily know the details of the relationship beforehand, and I want to give the parameters associated with the constraint to the optimizer so that it can update them during training. I am envisioning something (very loosely) like this:</p>
<pre><code class="lang-auto">def constraint(y,param_mat):
    return torch.mm(param_mat,y)

param_mat = torch.Parameter(torch.tensor(np.random.normal(0,1,(y_size,y_size)),requires_grad = True)

for i in range(num_nets):
    ipt = batch[0][i]
    target = batch[1][i]
    y = model_list[i](ipt)
    loss = loss1(y,target)+loss2(constraint(t,param_mat))
    optimizer.zero_grad()
    loss.backwards()
    optimizer.step()
</code></pre>
<p>Will this type of setup cause the parameter to be added to the computation graph and updated by the optimizer?</p>

ptrblck: <p>You would have to add <code>param_mat</code> to an optimizer (or create a new one for it) so that it can be optimized.</p>

algol: <p>So perhaps something like this?</p>
<pre><code class="lang-auto">def constraint(y,param_mat):
    return torch.mm(param_mat,y)

param_mat = torch.Parameter(torch.tensor(np.random.normal(0,1,(y_size,y_size)),requires_grad = True)
param_list = [list(model_list[i].parameters()) + list(params_mat) for i in range(num_models)]
optimizers_list = [optim.Adam(param_list[i]) for i in range(num_models)]
for i in range(num_nets):
    ipt = batch[0][i]
    target = batch[1][i]
    y = model_list[i](ipt)
    loss = loss1(y,target)+loss2(constraint(t,param_mat))
    optimizers_list[i].zero_grad()
    loss.backwards()
    optimizers_list[i].step()
</code></pre>

ptrblck: <p>This approach would pass the same <code>param_mat</code> to all optimizers, which would update it with its gradient and I assume that’s the use case you are looking for.</p>
Backward error in ddpg algorithm

Antonio_Niro: <p>Hello, I’m trying to implement a DDPG, but every time I try to update my networks, I have this error:<br>
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation. Can you help me to find the inplace operation that cause it?<br>
This is my code</p>
<pre><code class="lang-auto">batch = self.memory.extract_batch(self.batch_size)
                    states = []
                    actions = []
                    next_states = []
                    rews = []
                    dones = []

                    for exp in batch:
                        states.append(torch.FloatTensor(exp.state).unsqueeze(0))
                        actions.append(torch.FloatTensor(exp.action).unsqueeze(0))
                        next_states.append(torch.FloatTensor(exp.next_obs).unsqueeze(0))
                        rews.append(torch.tensor([exp.reward], dtype=float).unsqueeze(0))
                        dones.append(torch.tensor([exp.done], dtype=float).unsqueeze(0))

                    states = torch.cat(states)
                    actions = torch.cat(actions)
                    next_states = torch.cat(next_states)
                    rews = torch.cat(rews)
                    dones = torch.cat(dones)

                    next_inputs = next_states
                    inputs = states

                    next_actions = self.target_policy.forward(next_inputs)
                    target_qvals = self.target_qf.forward(next_inputs, next_actions.detach())
                    y = rews + (1.0 - dones) * self.discount * target_qvals
                    qval = self.qf.forward(inputs, actions)
                    qval_loss = (qval - y).pow(2).mean()
                    actions = self.policy.forward(inputs)
                    action_loss = -1 * self.qf.forward(inputs, actions).mean()

                    # optimize qf
                    self.qf_optimizer.zero_grad()
                    qval_loss.backward(retain_graph=True)
                    self.qf_optimizer.step()

                    # optimize policy
                    self.policy_optimizer.zero_grad()
                    action_loss.backward()
                    self.policy_optimizer.step()

</code></pre>
<p>Thank you in advance!</p>

ptrblck: <p>You might be trying to calculate the gradients using stale forward activations with already updated parameters as described in <a href="https://discuss.pytorch.org/t/struggling-with-a-runtime-error-related-to-in-place-operations/102222/6">this post</a>.</p>
Install Previous Nightly build with Pip

Diego: <p>I would like to install a previous nightly build for PyTorch using PIP, especifically, this version: <code>1.0.0.dev20190328</code>. Which is the one that gets installed when running: <code>conda install -c pytorch pytorch-nightly</code>.</p>
<p>I tried using: <code>pip3 install torch_nightly -f https://download.pytorch.org/whl/nightly/cu100/torch_nightly.html</code> but this, of course, install the latest nightly build (<code>torch_nightly-1.2.0.dev20190805</code>). How can I accomplish this without using conda.</p>

dickson_neoh: <p>Have you found any solution to this?</p>

ptrblck: <p>You could check the files from <a href="https://download.pytorch.org/whl/nightly/cu110/torch_nightly.html">here</a> and check, if the desired nightly can be found. Alternatively you could probably also specify the version directly (haven’t checked it using <code>pip</code> as I’m usually using <code>conda</code>).</p>
Custom Segmentation Model for Transfer Learning

AlperEkmekci: <p>I’ve trained a <strong>1 class</strong> UNet model that can be detect car shapes for segmentation properly, however I need a <strong>10 class</strong> classification process with the same dataset. I’m getting good results from <strong>only segmentation</strong>, so I’m trying to implement this model allows to make classification too.</p>
<p>I labeled my first dataset as I need to, so is there any way to merge this labeled dataset and 1 class segmentation model together and train them properly ?</p>

ptrblck: <p>Some models such as <code>MaskRCNN</code> output segmentation masks as well as classification results for the objects, so you might take a look at these implementations and check, if you could use a similar workflow for your use case.</p>

ccoding: <p>You can get two outputs by insert an output from your encoder of UNet, and fully connect them , add two or three dense layers, and output 10 classes as you want.</p>

AlperEkmekci: <p>Thanks for your replies, I really appreciate that.</p>
<blockquote>
<p>You can get two outputs by insert an output from your encoder of UNet, and fully connect them , add two or three dense layers, and output 10 classes as you want.</p>
</blockquote>
<p>Freeze the whole network and train the dense layers which is added to end of UNet ?</p>
<blockquote>
<p>Some models such as <code>MaskRCNN</code> output segmentation masks as well as classification results for the objects, so you might take a look at these implementations and check, if you could use a similar workflow for your use case.</p>
</blockquote>
<p>My labels and UNet class parameter is 1, because of that I get only background segmentation. I wonder which one is the more logical approach then the other one; add a layer to end of the network, freeze except these dense layers and train them only to get more classes or training same model with more classes with multi-labeled dataset ?</p>
How to do atomicAdd() on element returned by PackedTensorAccessor32

jvhoffbauer: <p>I have a cuda kernel taking the argument: <code>torch::PackedTensorAccessor32&lt;scalar_t, 4, torch::RestrictPtrTraits&gt; input</code></p>
<p>How do I perform atomic add on the elements of <code>input</code>?</p>
<p>I.e. <code>input[n][c][y][x] += (scalar_t) 1</code> works, but <code>atomicAdd(&amp;input[n][c][y][x], 1)</code> does not. The problem is, that it adds 256 instead of 1 to the tensor at the specified position. Thus I assume, that there is a problem with type conversions.</p>
<p>How can I do atomicAdd properly?</p>

albanD: <p>cc <a class="mention" href="/u/ptrblck">@ptrblck</a> that is more used to cuda primitives.</p>

ptrblck: <p>Could you pass the value as <code>scalar_t</code> or post a small code snippet to reproduce this issue, please?</p>

jvhoffbauer: <p>It works now. There was another mistake on my side.</p>

xieshuqin: <p>I ran into the same problem. The bug is very likely that <code>input</code> tensor is on <code>cpu</code> rather than <code>gpu</code>.</p>
<p>In my case, I define my output tensor using <code>torch::zeros()</code> rather than <code>torch::zeros_like()</code>, then my output tensor is on cpu and will not be modified by GPU functions.</p>
<p>In fact, an <code>illegal memory access</code> exception is thrown, but we can’t see it unless we explicitly synchronize after the kernel launch.</p>
<p>Add synchronizing code after kernel launching is very useful for my debugging</p>
<pre><code class="lang-auto">cudaError_t cudaerr = cudaDeviceSynchronize();
if (cudaerr != cudaSuccess)
    printf("kernel launch failed with error \"%s\".\n", cudaGetErrorString(cudaerr));
</code></pre>
<p>It makes the error message much more meaningful.</p>
DDP w/ multi-CPU: Sending different layers to different devices

ajayp1: <p>It looks like DDP for CPUs has been <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/parallel/__init__.py" rel="noopener nofollow ugc">deprecated</a>, so I assume that distributing training across multiple CPUs is virtually same as demonstrated in <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" rel="noopener nofollow ugc">this tutorial</a>, however I’m confused how to specify to send a model to a specific rank CPU.</p>
<p>That is, in the definition of ToyMpModel in the above tutorial, different layers of the model are sent to different GPUs via the .to() method. In this code, the .to() method simply takes an integer as argument. When one only has multiple CPUs and no GPUs, how does .to() know to send the layer to the rank CPU specified by the input to .to()?</p>
<p>When I try to input different integers to .to() on my local machine (which has a gpu), .to() automatically assumes that I’m sending a model/layer to a CUDA device - even when I specify <code>export CUDA_VISIBLE_DEVICES=""</code> before I run my code.</p>
ColorJitter Transformation for 16-bit images?

Vivek_Reddy: <p>As part of preprocessing for my image dataset consisting of 16-bit tiff images, I would like to perform a color jitter transformation.  I don’t want the network to be too sensitive to brightness in the image.  It seems, however, that torchvision.transforms.ColorJitter requires a PIL image, and PIL images must be a uint8 from [0,255].  Is this correct?  Should I use skimage or some other libraries to change brightness or is there a way I can get ColorJitter transform to work for 16 bit images in the range [0,65535].  Thank you!</p>
How to solve a installation problem

Seungsu: <p>Hi.</p>
<p>I am installing the pytorch by pytorch source code.</p>
<p>when i enter the command “python setup.py install”</p>
<p>occur [permission denied]</p>
<p>so i add “sudo python setup.py insall”</p>
<p>this command works well.</p>
<p>but problem is that pytorch package are installed in “/usr/local/lib/python3.7/dist-packages”</p>
<p>I want to change the installed directory to “~/.local/lib/python3.7/site-packages”</p>
<p>What i have to do?</p>
DNN Cannot Stop Overfitting

shawnbzhang: <p>I am training a DNN (CNN + RNN) for a voice conversion task. Although my train loss can be very low with good performance, I believe I am experiencing massive overfitting. To overcome this, I have already added quite a bit of batch norm and dropout inside the model as well as weight decay — however, the model still continues to overfit a lot. I present some of my loss curves below:</p>
<p>With a weight decay constant of 1e-7:<br>
<a href="https://i.stack.imgur.com/bUZ5T.png" rel="noopener nofollow ugc"><img src="https://i.stack.imgur.com/bUZ5T.png" alt="With a weight decay constant of 1e-7" width="690" height="247"></a></p>
<p>With a weight decay constant of 1e-2:<br>
<a href="https://i.stack.imgur.com/Xjlwv.png" rel="noopener nofollow ugc"><img src="https://i.stack.imgur.com/Xjlwv.png" alt="With a weight decay constant of 1e-2" width="690" height="251"></a></p>
<p>Note that I noticed that if the weight decay constant is &gt; 1e-4, the model seems to experience underfitting.</p>
<p>I want to know what else can I do to improve this model’s generalization. Is it just a matter of more data, or do I need to modify my DNN architecture in some way. I have been struggling with this overfitting problem for some days now, and any insight would be a help.</p>

Abhilash_Srivastava: <ol>
<li>Make sure your data is split appropriately between the train and the validation set. Use shuffle with stratified sampling if possible.</li>
<li>Experiment with learning rate (lr). The spikes in your dev_mse curve could be due to lr.</li>
<li>Which optimizer are you using? Adam or RAdam optimizer should work well in most cases.</li>
<li>Since you’ve already used experimented with dropout, you can also try using L1/L2 regularizations.</li>
</ol>
Check if A Tensor is optimized in Pytorch?

TinNguyenqn: <p>Hi, I have define a model like this:<br>
In model.py<br>
class Model(torch.nn.Module):<br>
def <strong>init</strong>(self, args):<br>
super(Model, self).<strong>init</strong>()</p>
<pre><code>    # Image Processing
    self.conv1 = nn.Conv2d(3, 128, kernel_size=8, stride=4) 
    self.conv2 = nn.Conv2d(128, 64, kernel_size=4, stride=2)
    self.conv3 = nn.Conv2d(64, 64, kernel_size=4, stride=2)

    # Initializing weights
    self.apply(weights_init)
    self.train()

def forward(self, inputs):
    """  
    encoder is the Encoder part of a AutoEncoder
    """
    encoder, x = inputs
    #get the auto encoder representation  
    ae_v_emb = encoder.view(encoder.shape[0], -1)
    
    # Get the image representation
    x = F.relu(self.conv1(x))
    x = F.relu(self.conv2(x))
    x_image_rep = F.relu(self.conv3(x))
    x_emb = x_image_rep.view(1, -1)
    
    #Concat features
    x_emb = torch.cat((x_emb, ae_v_emb), 1)
    
    return x_emb
</code></pre>
<p>In train.py<br>
I have defined training process like this:<br>
from model import Model<br>
model = Model()<br>
auto_encoder_model = AutoEncoder()<br>
optimizer = optim.SGD(model.parameters(), lr=args.lr)<br>
…<br>
<span class="hashtag">#run</span> Auto Encoder<br>
encode = auto_encoder_model.encoder(image)<br>
decode = auto_encoder_model.decoder(encoder)<br>
features = model(encode, image)<br>
…<br>
optimizer.step()</p>
<p>I have passed an RGB image and an encoder of an Auto Encoder model into this model. My question is that I don’t define an optimizer for the Auto Encoder model, so can the encoder part of the auto encoder model be optimized through the optimizer of the main model (Model) ?<br>
Thank you very much</p>

Abhilash_Srivastava: <aside class="quote no-group" data-username="TinNguyenqn" data-post="1" data-topic="108872">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tinnguyenqn/48/31587_2.png" class="avatar"> TinNguyenqn:</div>
<blockquote>
<p>optimizer = optim.SGD(model.parameters(), lr=args.lr)</p>
</blockquote>
</aside>
<p>You are passing the <code>model</code> parameters into the optimizer. So, the <code>model</code> parameters would be optimized (using SGD) and not the <code>auto_encoder_model</code> parameters. Unless there are pieces of code which are missing here.</p>

TinNguyenqn: <p>Thank you. So, how to optimizer AE parameters along with the Model?</p>
Extracting multiple sub tensors from a tensor simultaneously

sean_zhuh: <p>Hi, Let’s say we have an input tensor of shape <code>C x H x W</code>, and an index tensor of shape <code>N x 2 x 2</code>, where <code>N</code> is the number of sub tensors I want to extract from input tensor, each <code>index[i, :, :]</code> is a 2 x 2 matrix where the 1st row specifies the starting point and distance along dim <code>H</code>, and the 2nd row along dim <code>W</code>, a minimal working example is as follows:</p>
<pre><code class="lang-auto">input = torch.Tensor(C, H, W)
index = torch.LongTensor(N, 2, 2)
subs = []
for i in range(N):
    h_s, h_dis = index[i, 0, 0], index[i, 0, 1]
    w_s, w_dis = index[i, 1, 0], index[i, 1, 1]
    subs.append(input[:, h_s: h_s + h_dis, w_s: w_s + w_dis])
</code></pre>
<p>I’d like this indexing to be paralleled so that the <code>for loop</code> can be eliminated because I’ll do some compute-intensive stuff inside the <code>for loop</code> after I extract a sub tensor, any idea on how to do it? Besides, each sub tensor can be of varying size, so I need a list to store sub tensors, instead of concatenation since it requires dimension should be equal.</p>
Using 'DistributedDataParallel' occurs a weird error

shaoming20798: <p>When I use ‘DataParallel’, the codes work fine. But when I convert to ‘DistributedDataParallel’, something weird happens:</p>
<blockquote>
<p>File “/home/20798/HTCN/lib/model/faster_rcnn/vgg16_HTCN.py”, line 145, in forward<br>
x2_bn = self.bn2(x2_fc)</p>
</blockquote>
<p>And the codes <code>File "/home/20798/HTCN/lib/model/faster_rcnn/vgg16_HTCN.py", line 145, in forward</code> metioned above is as follows:</p>
<pre data-code-wrap="python"><code class="lang-python">class netD_da(nn.Module): # line 129
    def __init__(self, feat_d):# line 130
        super(netD_da, self).__init__()# line 131
        self.fc1 = nn.Linear(feat_d,100)# line 132
        self.bn1 = nn.BatchNorm1d(100)# line 133
        self.fc2 = nn.Linear(100,100)# line 134
        self.bn2 = nn.BatchNorm1d(100)# line 135
        self.fc3 = nn.Linear(100,2)# line 136
    def forward(self, x):# line 137
        #x1 = F.dropout(F.relu(self.bn1(self.fc1(x))),training=self.training)# line 138
        x1_fc = self.fc1(x)# line 139
        x1_bn = self.bn1(x1_fc)# line 140
        x1_relu = F.relu(x1_bn)# line 141
        x1 = F.dropout(x1_relu)# line 142
        #x2 = F.dropout(F.relu(self.bn2(self.fc2(x1))),training=self.training)# line 143
        x2_fc = self.fc2(x1)# line 144
        x2_bn = self.bn2(x2_fc)# line 145
        x2_relu = F.relu(x2_bn)# line 146
        x2 = F.dropout(x2_relu)# line 147
        ret = self.fc3(x2)# line 148
        return ret  #[256, 2]# line 149
</code></pre>

pritamdamania87: <p><a class="mention" href="/u/shaoming20798">@shaoming20798</a> Can you provide a minimal repro for this issue?</p>
Detect model size

whoab: <p>Let’s say we have 10 neural network models. In order to decide how to partition these models amongst a set of GPUs, we need to know the size of each model. How do we calculate that?</p>

ptrblck: <p>You can estimate the memory footprint of the model itself by summing the number of parameters, buffers (, and other tensors, if needed) and multiply it by the <code>dtype</code> factor (e.g. 4 for <code>float32</code>).<br>
However, this would not give you the “complete” memory usage, since the forward activations (intermediates) as well as the gradients would also use memory. That being said, if you are using cudnn, then note that different algorithms would also consume a different amount of memory for their workspace, especially if you are using <code>torch.backends.cudnn.benchmark = True</code>.<br>
I would thus recommend to perform an example training step with the shapes you are planning to use and check the memory usage e.g. via <code>torch.cuda.memory_summary()</code>.</p>

whoab: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="107482">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>I would thus recommend to perform an example training step with the shapes you are planning to use and check the memory usage e.g. via <code>torch.cuda.memory_summary()</code></p>
</blockquote>
</aside>
<p>The problem is usually these models I am referring to are too big to fit in memory, and the point of calculating the memory is we are deciding how to distribute each sub-model onto multiple GPUs. So, I can’t load the entire model first to calculate the size.</p>
<p>Do you have any code for the first method of calculating memory?</p>
Launch independent pytorch processes on each GPU from python

adityamnk: <p><strong>Problem:</strong><br>
I would like to launch one pytorch task per GPU from within python. Each process is independent and does not need to share any data.</p>
<p><strong>Solution:</strong><br>
I use a similar program as shown below. Using ‘nvidia-smi’, I am able to verify that I am using all GPUs when I run the code below.</p>
<pre><code class="lang-auto">import os
from multiprocessing import Process
import time

def launch_proc(gpu_id):
    os.environ['CUDA_VISIBLE_DEVICES'] = str(gpu_id)
    import torch
    #Allocate memory on visible GPU
    tensor = torch.Tensor((10**8)*[gpu_id]).cuda()
    time.sleep(10) #wait for 10 seconds 

num_gpus = 4
processes = []
for gpu_id in range(num_gpus):
    p = Process(target=launch_proc,args=(gpu_id,))
    p.start()
    processes.append(p)

for p in processes:
    p.join()
</code></pre>
<p><strong>Question:</strong><br>
Is there a cleaner implementation? Or, is the above solution the best solution? In the above solution, if I use torch.cuda.device_count() to get the number of gpus, I get the following error -<br>
<code>RuntimeError: CUDA error: initialization error</code></p>

imesery: <p>hey, I’m having a similar use-case.<br>
what did you do eventually?<br>
Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

adityamnk: <p>I used something similar to the following code -</p>
<pre><code class="lang-auto">import os
import multiprocessing as mp
import time
import torch

def launch_proc(gpu_id):
    os.environ['CUDA_VISIBLE_DEVICES'] = str(gpu_id)
    import torch
    #Allocate memory on visible GPU
    tensor = torch.Tensor((10**8)*[gpu_id]).cuda()
    time.sleep(10) #wait for 10 seconds 

if __name__ == '__main__':
    num_gpus = torch.cuda.device_count()
    ctx = mp.get_context('spawn')
    with ctx.Pool(processes=num_gpus) as pool:
        pool.starmap(launch_proc, [(i,) for i in range(num_gpus)])

</code></pre>
<p>The main change was to use the <code>spawn</code> context and enclose the GPU launch processes within <code>if __name__ == '__main__'</code>.</p>
F.softmax same input but different output

woong97: <p>Is it possible? F.softmax returns slightly different outputs even though inputs are same.</p>
<p>And if each tensor of input1 and input2 are different with almost 0.00001 then will softmax be different quitely?</p>

ptrblck: <p>You might see rel. small errors due to the limited floating point precision, if the used algorithm is non-deterministic.<br>
Could you set <code>torch.set_deterministic(True)</code> and rerun the script? Also, which PyTorch version are you using and CPU/GPU for this test?</p>
Gradients not computed after replacing BatchNorm with GroupNorm

phaedrus: <p>I’m fine-tuning Resnet18, and wanted to try replacing BatchNorms with GroupNorms to see if I can do smaller batches, but somehow, none of the weights in the network are updating when I replace BatchNorms with GroupNorms.</p>
<p>Here is my code to replace BatchNorms:</p>
<pre><code class="lang-auto">def convert_bn_model_to_gn(module, num_groups=16):
    mod = module
    if isinstance(module, torch.nn.modules.batchnorm._BatchNorm):
        num_groups = 1
        mod = torch.nn.GroupNorm(num_groups, module.num_features,
                           eps=module.eps, affine=module.affine)
        # mod = torch.nn.modules.linear.Identity()
        if module.affine:
            mod.weight.data = module.weight.data.clone().detach()
            mod.bias.data = module.bias.data.clone().detach()
    for name, child in module.named_children():
        mod.add_module(name, convert_bn_model_to_gn(child, num_groups=num_groups))
    del module
    return mod
</code></pre>
<p>Inspecting the parameters (even parameters of the dense layer at the top, right before the loss), the gradients are zeros (after calling <code>loss.backward()</code>), and the weights don’t change. When I keep the BatchNorms, it works fine. Even when I replace BatchNorms with BatchNorms, it works.</p>
<p>Somehow, when replacing BNs with GNs, it disconnects the graph or something and doesn’t computer gradients. Any ideas what I’m doing wrong? Or ideas how to debug?</p>
Application of activation function (not yet finally answered)

THEblob: <p>Edit:  Current question in my last post down below!</p>
<p>Is this function correct or does it apply<br>
the relu to the input layer and is therefore wrong?</p>
<blockquote>
<p>def forward(self, input):<br>
x = self.fc1(input)<br>
x = nn.functional.relu(x)<br>
x = self.fc2(x)<br>
x = nn.functional.relu(x)<br>
x = self.fc3(x)<br>
output = nn.functional.relu(x)<br>
return output</p>
</blockquote>
<p>Sry if this is too basic but I’m still new to machine learning.<br>
An advice would be much appreciated. Gomen!</p>

googlebot: <p>input part is ok, but final relu may be bad - you won’t produce negative outputs, needed/adequate for some losses/targets.</p>

THEblob: <p>Thank you very much.<br>
I think what confuses me is the concept of layer. When I implement the first layer of neurons I set a number for it’s inputs and it’s outputs. So is the input layer only the first part of it and the outputs belong already to the hidden layer and not the input layer anymore? That’s the point that made me think I might applied the activation function to the input layer.</p>
<p>Edit: So my code example above has in fact two hidden layers not one am I right? The self.fc() just implements the connection between two layers?</p>

googlebot: <p>You have 3 trainable transformations (fcN), if this function defines a whole network, you may call them input layer, hidden layer and output layer, otherwise you may call this a block to disambiguate. You can set fc1,fc2 output sizes independently and arbitrary as they correspond to internal data representations. Two relu activations in between can be viewed as separators that provide non-linearity.</p>

THEblob: <p>Ok, thank you. To come back to my initial question with the activation function,<br>
if fc1 is indeed my input layer why is the application of relu still fine? I’ve read that input layers don’t have an activation function only hidden and output layers do. I hope it’s clear now what my point of confusion is.</p>
<p>Edit: So basically this is it of what I don’t get why it still should be fine:</p>
<blockquote>
<p>x= self.fc1(input)<br>
x = nn.functional.relu(x)</p>
</blockquote>

googlebot: <p>That would be the case if the author logically groups fc transformations with preceding activations; there is some abiguity about treating activations as belonging to one of its neighbours.<br>
Basically, rule about input is - don’t distort input by activations before you apply the first trainable transformation.</p>

THEblob: <p>Great thanks to you, this gave me clearity! <img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=9" title=":+1:" class="emoji" alt=":+1:"></p>