akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Implementing quantile loss

username: Void
cooked: <p>Hi,</p>
<p>I’m wondering if there is an established way to use quantile loss in Pytorch? I’d like to build a network that outputs several quantiles at the same time when making a prediction.</p>
<p>How would I go about it?</p>
<p>Thanks</p>

username: googlebot
cooked: <p>check links in <a href="https://github.com/pytorch/pytorch/issues/38035" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/issues/38035</a>. I haven’t used these, but experimented with “asymmetric laplace distribution” and “huber quantile loss” instead, the latter one has varying gradients instead of {-1,+1} and worked better from what I recall.</p>

username: Void
cooked: <p>I’ve looked at it as well as the <a href="https://pytorch-forecasting.readthedocs.io/en/latest/api/pytorch_forecasting.metrics.QuantileLoss.html?highlight=quantile" rel="noopener nofollow ugc">pytorch-forecasting</a> implementation but I’m not sure I get it because there’s manipulation for the shapes which don’t make sense to me such as unsqueezing the result.</p>

username: googlebot
cooked: <p>hmm, yes, it is tricky, the network should produce multiple ordered predictions, but you have only one observation for particular inputs (in usual case), so you unsqueeze the target. And as quantile specific gradients will still be different, prediction producing layer will eventually learn to output reasonable quantile grids, conditioned on input.</p>
Register_forward_hook after every n steps

username: shuxp
cooked: <p>I am training my own code, and I want to implement one function:</p>
<p>After training n(=100) iterations, I want to see feature maps.</p>
<p>Now I have done with register_forward_hook to see the feature maps, But I want to see them after n iterations. Any idea about how to implement this?</p>
<p>Thanks.</p>

username: shuxp
cooked: <p>I have implement it already…but a bit slow</p>

username: mailcorahul
cooked: <p>You can keep track of the <code>step</code> or <code>iteration</code> by using a member variable inside the Network module or the module containing the hook function.<br>
For example, inside your Network module,</p>
<pre><code class="lang-auto"># network module
class Network(nn.Module):
    def __init__(self):
        self.step = -1
# while training
model = Network()
for i in range(steps):
    # increment step var before forward pass
    model.step += 1
    output = model(input) # check self.step inside register_forward_hook
</code></pre>

username: ptrblck
cooked: <p>That’s an interesting approach!<br>
Based on this, you could also create a “hook class” holding this internal parameter, which could be simpler than creating new modules:</p>
<pre><code class="lang-python">class ActivationHook():
    def __init__(self, name, every_steps=1):
        self.name = name
        self.every_steps = every_steps
        self.steps = 0
        self.activation = {}
    
    def hook(self, model, input, output):
        if (self.steps % self.every_steps) == 0:
            self.activation[self.name] = output.detach()
        self.steps += 1
    

model = MyModel()
act_hook = ActivationHook('fc2', every_steps=2)
model.fc2.register_forward_hook(act_hook.hook)
x = torch.randn(1, 25)
output = model(x)
print(act_hook.activation['fc2'])
</code></pre>

username: Rio1210
cooked: <p>This does not seem to work!</p>
Backpropagation not the same time complexity as forward pass in simple MLP with views/slicing

username: outlace
cooked: <p>In a simple multilayer perceptron / fully connect NN, the forward pass is just a sequence of matrix multiplications composed with some nonlinearities. So the time complexity of the forward pass is basically just the time complexity of the matmul operations, which is polynomial time O(n^2.3) or something. The backward pass should have the same time complexity as the forward pass since it also consists of basically the same matmul operations.</p>
<p>I am working on implementing a technique for approximate matrix multiplication where I adaptively downsample the weight matrices of each layer of an MLP such that, no matter how big the layer weight matrices are, the down-sampled matrices will remain relatively constant or at most grows sub-linearly with the size of the original matrices.</p>
<p>Here’s a minimally working example:</p>
<pre><code class="lang-python">import torch
import cProfile

def test1(x,W):
    return torch.sum(x @ W)

def test2(x,W,S):
    return torch.sum(x @ W[:,S])

dim1,dim2 = 784,60000
x = torch.randn(1,dim1);
W1 = torch.randn(dim1,dim2,requires_grad=True)
W2 = torch.randn(dim1,dim2,requires_grad=True);
S = torch.randint(0,dim2,(50,));

cProfile.run("test1(x,W1)") #0.003 seconds -&gt; 0.009 -&gt; 0.016, appears to be scaling superlinearly
cProfile.run("test2(x,W2,S)") #0.006 seconds -&gt; 0.005 -&gt; 0.006, runs in constant time as expected
cProfile.run("test1(x,W1).backward()") # 0.016 seconds -&gt; 0.078 -&gt; 0.144, scaling superlinearly
cProfile.run("test2(x,W2,S).backward()") #0.023 seconds -&gt; 0.065 -&gt; 0.220, also runs about as slow as test1
</code></pre>
<p>Why is the backward pass of test2 running as slowly as test1 and scaling superlinearly when I expect it to run in constant time like the forward pass of test2?</p>
Explanation to MaxPool2d

username: tomera
cooked: <p>Hi, I need helping understating what MaxPool2d function does.<br>
I read and the docs and all the example but still Im not sure about it.<br>
For the docs:</p>
<blockquote>
<p>“Applies a 2D max pooling over an input signal composed of several input planes.”</p>
</blockquote>
<p>What is “max pooling” and why do we need it?</p>
<p>Thanks.</p>

username: martinr
cooked: <p>Well, you can probably get a getter answer from any machine learning book, but in essence it scales down the matrix. You will still have the same data, but scaled down version. How much it scales is determined by stride parameter.</p>
<p>Kernel size (a small window to look) determines the area to “pool” over and stride determines the step. Imagine, it starts applying kernel from upper left corner and moves the kernel by stride. So kernel 1x1 and stride 1 does nothing, keeps the input. Kernel 2x2, stride 2 will shrink the data by 2. Shrinking effect comes from the stride parameter (a step to take). Kernel 1x1, stride 2 will also shrink the data by 2, but will just keep every second pixel while 2x2 kernel will keep the max pixel from the 2x2 area.</p>
<p>You can also achieve the shrinking effect by using stride on conv layer directly. Difference is that conv layer also learns while pooling layer does not. There is no learning happening on the pooling layer.</p>

username: tomera
cooked: <p>Thanks for the detailed answer.</p>
<p>In the course in ML I took they didn’t mentioned pool size or shrinking techniques. They did mention mini batches and other shirking methods for linear and logistic regression. Is this the equivalent for NN?<br>
If we don’t define pool what will happen? It’s only for performances right? What is the motivation to shrink the data?</p>

username: martinr
cooked: <p>This answer perhaps comes too late for you, but the major use case of pooling is that when you shrink the data then a convolutional layer will learn higher level features (i.e. pooling aggregates information).  E.g., your (few) first high-dimensional layer(s) will learn small edges, curves etc; your pooled layer will learn lines and shapes; and finally further pooled layer recognizes whole objects. So it is not only for performance, but to actually learn <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
How does batch norm behave in pytorch depending if one is in training mode or eval/inference mode?

username: Brando_Miranda
cooked: <p>I was looking at:</p>
<p><a href="http://pytorch.org/docs/stable/_modules/torch/nn/modules/batchnorm.html#BatchNorm2d" class="onebox" target="_blank" rel="nofollow noopener">http://pytorch.org/docs/stable/_modules/torch/nn/modules/batchnorm.html#BatchNorm2d</a></p>
<p>and I didn’t see how how F.batch_norm works. I looked in the docs but could find it and perhaps it seems that it might be written in C which I dont know.</p>
<p>My question is how does batch_norm updates its parameters during training vs inference? When are the mean variance for example updated? What happens to them during inference mode? Are they just fixed?</p>

username: Brando_Miranda
cooked: <aside class="quote" data-post="1" data-topic="16857">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/brando_miranda/48/14355_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-does-pytorch-s-batch-norm-know-if-the-forward-pass-its-doing-is-for-inference-or-training/16857">How does pytorch’s batch norm know if the forward pass its doing is for inference or training?</a> 
  </div>
  <blockquote>
    how does pytorch’s batch norm know if the forward pass its doing is for inference or training? I am evaluating the the test performance of my net but realized that I’m not sure how my net knows if its training phase or inference phase. How does pytorch handle this?
  </blockquote>
</aside>
Can't pass scriptModule as const parameter

username: armony
cooked: <p>When I try to pass scripted module as const reference I get this error in MSVC:</p>
<pre><code class="lang-auto">the object has type qualifiers that are not compatible with the member function "torch::jit::Module::forward"
</code></pre>
<p>What is the cause of this?<br>
Any help?</p>
How to use _forward_impl in my own model

username: hotpanda
cooked: <p>I used torch.onnx.export to convert my model to onnx. My Module that is not  a <code>ScriptModule</code>. that means the exported model will <em>not</em> capture this dynamic behavior,<br>
so when I ran inference with the exported onnx model, I got the wrong predictions.<br>
Using ScriptModule to rebuild my model is a huge work. I converted resnet50 to onnx, and can get the right output, the Resnet in pytorch is torch.nn.Module, I want to know how Resnet worls fine in onnx, and what’s the meaning of _forward_impl, and how to use _forward_impl.</p>
Stratified Multilabel Dataset in Pytorch

username: xdwang0726
cooked: <p>Is there any method in Pytorch like train_test_split in sklearn to stratify the dataset into train, valid and test for multilabel classifications?</p>

username: ptrblck
cooked: <p>No, I don’t think there is a built-in PyTorch method, but you could of course use the scikit-learn methods and I would also recommend to use already implemented and tested methods from the Python ecosystem, if possible.</p>

username: xdwang0726
cooked: <p>Okay, thanks for your reply!</p>
Hello World aka. MNIST with feed forward gets less accuracy in comparison of plain with DistributedDataParallel (DDP) model with only one node

username: Joerg_Simon
cooked: <p>Dear PyTorch Community,</p>
<p>When rewriting my library for training I stumbled upon the following problem: When I train with DistributedDataParallel (DDP) I get less accuracy than without. Especially if I limit DDP to only one node and one gpu I would expect that DDP and non-DDP gives approximately the same result given the same code and Hyperparameters. It does not for me.</p>
<p>I distilled the problem in two examples of the Hello World for DeepLearning: Training a super simple FeedForward Network on MNIST. It has exactly the same hyperparemeters for distributed and not distributed training and generally the same code beside the adoptions needed to use the DDP library. Also all seeds are fixed to make it reproducible. The most astonishing things is: When I train <strong>the DDP model with only one node</strong> it still gives about 10% less accuracy. I tried this with different pytorch versions (1.4, 1.8.1 and 1.9.0) on four different computers (MacBook Pro on CPU, three different ubuntu machines, one with a 1080-ti, one with a 2080-ti and a Cluster with P100s inside). So no matter os OS or GPU or CPU when I limit DDP to one node only and compare it to the plain training I get an extremely worse accuracy. Also if I f.e. use two nodes and double the learning rate with the same batch size it gives me this worse results.</p>
<p>Now I wonder how that could be. I can not imagine I am the only one to stumble upon that problem, but if it is a user error, than it would be cool to see what it is. I attached a link to a GitHub repo with both scripts.</p>
<p>This is a link to the repository with the example: <a href="https://github.com/joergsimon/mnist-distributed-problem" class="inline-onebox" rel="noopener nofollow ugc">GitHub - joergsimon/mnist-distributed-problem: This is a super small repository demonstrating a Problem with DistributedDataParallel. A three layer feed forward neural network is trained with MNIST with and without data parallel with the same hyper parameters. If you configure DistributedDataParalell to use only one node, the model is quite worse in accuracy. If you have any suggestions how to make them equal beside tuning the learning rate please commend or send PR!</a></p>
<p>Any suggestions would be awesome!</p>

username: Joerg_Simon
cooked: <p>someone answered the question on stack overflow. Basically over trying things out apparently a different batch_size slipped into the two different versions. Correcting that gives the same result… <a href="https://stackoverflow.com/questions/69100602/hello-world-aka-mnist-with-feed-forward-gets-less-accuracy-in-comparison-of-pla" class="inline-onebox" rel="noopener nofollow ugc">pytorch - Hello World aka. MNIST with feed forward gets less accuracy in comparison of plain with DistributedDataParallel (DDP) model with only one node - Stack Overflow</a></p>
Let WordEmbedding accept feature vectors

username: HallerPatrick
cooked: <p>Hello,</p>
<p>as far as I know is the WordEmbedding layer used to store/retrieve embeddings of words for a given indices (depending or embedding size and dict size).</p>
<p>So for example a word is represented as an index and a sentence as a list of indexes.</p>
<p>What if I don’t want to pass indices, but for example a one-hot encoding of my given word(s). So more or less to the encoding manually? I looked at the WordEmbedding class but couldn’t really figure out if subclassing WordEmbedding myself or if I should build my own embedding class (one-hot encoding + linear layer).</p>
<p>I hope it is clear, what I want to achieve.</p>
<p>Greetings,<br>
Patrick</p>

username: tom
cooked: <p>Hello Patrick,</p>
<p>you mean the Embedding layer?</p>
<p>If you have one-hot encoded words, you can use <code>torch.nn.functional.linear</code> (which is matrix multiplication, what happens in  a a Linear layer without bias) to get the same result, here with</p>
<pre><code class="lang-auto">e = torch.nn.Embedding(2, 5)

idx = torch.tensor([[0, 1]]) # batch x seq
print(e(idx)) # batch x seq x embedding

one_hot = torch.nn.functional.one_hot(idx, num_classes=2).float()

print(torch.nn.functional.linear(one_hot, e.weight.t()))
</code></pre>
<p>Best regards</p>
<p>Thomas</p>

username: HallerPatrick
cooked: <p>Hello Thomas,</p>
<p>great idea. Thank you for the solution!</p>
<p>Patrick</p>
How to store model in memory when serving/deploy

username: John_J_Watson
cooked: <p>I have this trained model (jit traced) now, and yes I can serve it using flask etc… but, every time the inference is done, the model is loaded each time - I would like to avoid this model load every time and keep this in memory somehow. What are the suggested approaches for this? I could not find any on pytorch site.</p>

username: Anakin
cooked: <p>Hello <a class="mention" href="/u/john_j_watson">@John_J_Watson</a>,</p>
<p>Perhaps you have already checked these links: <a href="https://pytorch.org/tutorials/intermediate/flask_rest_api_tutorial.html" rel="noopener nofollow ugc">DEPLOYING PYTORCH IN PYTHON VIA A REST API WITH FLASK</a>and <a href="https://github.com/avinassh/pytorch-flask-api-heroku" rel="noopener nofollow ugc">PyTorch Flask API</a>. Especially, the second one shows that the load_model function can be written somewhere else, not in the script where flask lauches directly.<br>
So the flask won’t reload the model again and again.</p>
<p>Hope it helps somehow.<br>
Jian</p>
How to use data augmentation to generate adversarial example

username: yangbo93
cooked: <p>In essence, the problem is how to reproduce the Pytorch version of MI-DI2-FGSM. As we all know, the original code of the MI-DI2-FGSM method is the TensorFlow version. Now I want to implement the Pytorch version of MI-DI2-FGSM. The difference between MI-FGSM and MI-DI2-FGSM is that MI-DI2-FGSM has an additional step of data augmentation. In MI-FGSM, the gradient is obtained from the original picture data, while in MI-DI2-FGSM, the gradient is obtained from the randomly transformed picture. Of course, only when calculating the gradient, the original picture is transformed immediately, and then the gradient is obtained for the randomly transformed picture. At other times, the original picture is not transformed randomly. However, I have reproduced the Pytorch version of MI-FGSM, but I have encountered difficulties in applying data augmentation technology to MI-FGSM. In Pytorch, the following methods are generally used for data augmentation: “transform = transforms.Compose([transforms.RandomHorizontalFlip(),transforms.ToTensor(), transforms.Normalize((0.0,), (1.0,))]); dataset = datasets.MNIST(root = ‘./data’, train=True, transform = transform, download=True)”, that is, the data augmentation step must be before transforms.Totensor(). However, this data augmentation method leads to random transformation not only when calculating the gradient, but also at other times. The result is that the generated adversarial examples = transformed image + perturbation, which is wrong. I hope you can provide ideas to solve this problem. Thank you.</p>
Uploading model into pytorch mobile

username: Gabriele_Ranzieri
cooked: <p>Hi,</p>
<p>I’ve uploaded my custom model into a pytorch mobile app with the code “module = Module.load(assetFilePath(this, “model_Im2Im_enc.pt”));”</p>
<p>I don’t know why but as soon as this line is processed, my app stop to work.</p>
<p>How can i fix it? I’ve uploaded some other model like MobileNetV2 and all is working good, the problem comes when i start using custom model.</p>
<p>Thanks in advice</p>
Different test accuracy with changed batch size in test set

username: Can_Nguyen
cooked: <p>Hi everyone!</p>
<p>I’ve just found out that when I increased the batch size value to calculate accuracy in my test set, the accuracy is changed. For example:</p>
<div class="md-table">
<table>
<thead>
<tr>
<th>BatchSize</th>
<th>Test Set 1</th>
<th>Test Set 2</th>
<th>Test Set 3</th>
<th>Test Set 4</th>
<th>Test Set 5</th>
<th>Test Set 6</th>
<th>Test Set 7</th>
</tr>
</thead>
<tbody>
<tr>
<td>512</td>
<td>99.60</td>
<td>99.428</td>
<td>95.628</td>
<td>96.8666</td>
<td>95.749</td>
<td>90.133</td>
<td>93.34</td>
</tr>
<tr>
<td>1024</td>
<td>99.63</td>
<td>99.428</td>
<td>95.628</td>
<td>96.8499</td>
<td>95.716</td>
<td>90.116</td>
<td>93.38</td>
</tr>
</tbody>
</table>
</div><p>Even <code>model.eval()</code> is all set, the accuracy has a small gap between each batch size value in some set.</p>
<p>Do you have any explanations for this?</p>
<p>Thanks,</p>
Local "slowdown" with Neural Style Transfer Tutorial

username: Andi
cooked: <p>Hello, I’m trying my first steps with the official Pytorch NST tutorial. Colab works great, also moving to Gatys’ layers, with Adam and l1 loss.<br>
At the moment I try to run it locally on Jupyter/Win10/GTX1650TI4GB it freezes at “Optimizing…” without any error. run[0] increases in the closure loop, if I print it.<br>
Any ideas what happens here? Did maybe anyone encouter that b4 - not understanding why that happens atm… Thanks in advance!</p>

username: Andi
cooked: <p>Ok, my bad, it actually IS running. But it runs really slowly. Is my card that bad? A memory shortage (it uses only 2.5GB of the 4…)? Any other read-write management to advise?<br>
Thanks for any suggestions! BR Andy</p>

username: gphilip
cooked: <p>Without seeing the code that you use, it is hard to diagnose what is wrong.</p>

username: Andi
cooked: <p>Hi, thank You for looking at. Basically the original code from the Pytorch NST Tutorial runs arond 5 times slower locally, colab+GPU vs. Local 1650TI, and I don’t get why.<br>
The official code is here <a href="https://pytorch.org/tutorials/advanced/neural_style_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Neural Transfer Using PyTorch — PyTorch Tutorials 1.9.0+cu102 documentation</a><br>
Thanks, br</p>

username: gphilip
cooked: <p>Did you try printing out <code>device</code> to verify that your local code is actually running on the GPU?</p>

username: Andi
cooked: <p>Yes, checked, cuda true, device active number, and name of this number, checkedbin system manager that card’s memory gets used (2.5GB of four). A google search for performance says - if I read it with “points” correctly - 1650ti should be 3-4 times faster than the tesla on colab…</p>

username: gphilip
cooked: <p>You could try finding the step that is taking the most time, using code similar to the following:</p>
<pre><code class="lang-auto">from time import time

start_time = time()
# Code suspected of being slow goes here  
print('{} elapsed time {}s'.format(time() - start_time))
</code></pre>
<p>Once you narrow the slowdown to a specific line, you can investigate it better.</p>

username: Andi
cooked: <p>Hello, I had a lot other tasks, so it took quite a while. Thanks again for looking into it.<br>
Having measured it, and cleaned up as much as possible, it seems not as bad as it was.<br>
Thnx again. BR.</p>
<p>Timings, fYi, on the same task (10000xNST, 512x512 image)<br>
Tesla K80 Colab: 3524 s (will be dependent on shared usage …)<br>
GTX1650TI 4 GB Notebook: 2230 s<br>
GTX1080TI 11 GB as eGPU over Thunderbolt 3: 580 s</p>
Memory Leak in WGAN-GP autograd

username: TimLiu
cooked: <p>I want to use WGAN-GP, and when I run the code, it gives me an error:</p>
<pre><code>def calculate_gradient_penalty(real_images, fake_images):

    t = torch.rand(real_images.size(0), 1, 1, 1).to(real_images.device)
    t = t.expand(real_images.size())

    interpolates = t * real_images + (1 - t) * fake_images
    interpolates.requires_grad_(True)

    disc_interpolates = D(interpolates)

    grad = torch.autograd.grad(
        outputs=disc_interpolates, inputs=interpolates,
        grad_outputs=torch.ones_like(disc_interpolates),
        create_graph=True, retain_graph=True, allow_unused=True)[0]

    grad_norm = torch.norm(torch.flatten(grad, start_dim=1), dim=1)
    loss_gp = torch.mean((grad_norm - 1) ** 2) * lambda_term

    return loss_gp
</code></pre>
<blockquote>
<p>RuntimeError                              Traceback (most recent call last)<br>
 in </p>
<p>/opt/conda/lib/python3.8/site-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph, inputs)<br>
243                 create_graph=create_graph,<br>
244                 inputs=inputs)<br>
 → 245         torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)<br>
246<br>
247     def register_hook(self, hook):</p>
<p>/opt/conda/lib/python3.8/site-packages/torch/autograd/<strong>init</strong>.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables, inputs)<br>
143         retain_graph = create_graph<br>
144<br>
 → 145     Variable.<em>execution_engine.run_backward(<br>
146         tensors, grad_tensors</em>, retain_graph, create_graph, inputs,<br>
147         allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag</p>
<p>RuntimeError: CUDA out of memory. Tried to allocate 64.00 MiB (GPU 2; 15.75 GiB total capacity; 13.76 GiB already allocated; 2.75 MiB free; 14.50 GiB reserved in total by PyTorch)</p>
</blockquote>
<p>Does anyone know how to solve this problem?</p>

username: ptrblck
cooked: <p>Could you check, if <code>prob_interpolated</code> has a valid <code>.grad_fn</code> attribute?<br>
If <code>print(prob_interpolated.grad_fn)</code> returns <code>None</code>, the model output seems to be detached from the computation graph and this error would be raised.</p>

username: TimLiu
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thx, I already solve this problem, but now seems the memory leak…Do you have any suggestions? The case it that, after every epoch, the memory increase…</p>

username: ptrblck
cooked: <p>In the <code>grad</code> call you are using <code>retain_graph=True</code>, so I would assume that the graph is not freed afterwards. Is this on purpose and if so, why do you need to retain the graph?</p>

username: TimLiu
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> In WGAN-GP, I think without it, the gradient (of the GP part of loss) cannot backproprogation through the entire D network, also there will be an error, RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn…</p>
Is there any func to do "select and pad" in PyTorch?

username: shihanmax
cooked: <p>Hey guys, I want to select tensor B along some dimension by some condition of A, specifically,</p>
<p>assume:<br>
A = [[0, 1, 1],<br>
[1, 3, 4],<br>
[2, 0, 0]] (3x3)</p>
<p>B = torch.randn(3x3x10)</p>
<p>I want to select dim 2 of B (1x1x10) by some condition of A, say, A[i,j] == 1,</p>
<p>mask = A==1:<br>
[[False, True, True],<br>
[True, False, False],<br>
[False, False, False]]</p>
<p>if mask[i,j] is True, select B[i, j, :], (and ‘move it to left’)</p>
<p>we will get:<br>
result=<br>
[[B[0, 1, :], B[0, 2, :]],<br>
[B[1, 0, :]]<br>
[]]</p>
<p>and add some padding:<br>
result=<br>
[[B[0, 1, :], B[0, 2, :], PAD],<br>
[B[1, 0, :], PAD, PAD]<br>
[PAD, PAD, PAD]]</p>
<p>I wrote a loop version like:</p>
<pre><code class="lang-auto">def select_and_pad(
    input_repr, ner_label_indices, indices, max_len,
):
    """Selecting tensors by specified ner indices along some axis.

    Args:
        input_repr (tensor): [bs, seq_len, hidden]
        ner_label_indices (tensor): [bs, seq_len]
        indices (list): specific entity indices
        max_entity_num (int): max seq len after padding

    Returns:
        selected_repr (tensor): [bs, max_len, hidden]
    """
    bs, seq_len, hidden_dim = input_repr.shape

    selected_repr = []

    for i in range(bs):
        curr = []
        
        for j in range(seq_len):
            curr_label = ner_label_indices[i][j].item()
            
            if curr_label in indices:
                curr.append(input_repr[i, j, :])

        # truncate
        if len(curr) &gt; max_len:
            curr = curr[:max_len]

        # pad zeros
        curr.extend([
            torch.zeros(hidden_dim) 
            for _ in range(max_len - len(curr))
        ])
        
        selected_repr.append(torch.stack(curr, dim=0))

    selected_repr = torch.stack(selected_repr, dim=0)

    return selected_repr


A = torch.tensor([[0,1,1], [1,3,4]])
B = torch.arange(1, 7).view(2,3).unsqueeze(2).repeat(1,1,10)

select_indices = [1]  # select B[i, j, :] where A[i, j] == 1

print(f"A: {A.shape}\n", A, "\n")
print(f"B: {B.shape}\n", B, "\n")

selection = select_and_pad(B, A, select_indices, max_len=3)

print(f"result: {selection.shape}\n", selection)
</code></pre>
<p>the result is:</p>
<pre><code class="lang-auto">A: torch.Size([2, 3])
 tensor([[0, 1, 1],
        [1, 3, 4]]) 

B: torch.Size([2, 3, 10])
 tensor([[[1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
         [2, 2, 2, 2, 2, 2, 2, 2, 2, 2],
         [3, 3, 3, 3, 3, 3, 3, 3, 3, 3]],

        [[4, 4, 4, 4, 4, 4, 4, 4, 4, 4],
         [5, 5, 5, 5, 5, 5, 5, 5, 5, 5],
         [6, 6, 6, 6, 6, 6, 6, 6, 6, 6]]]) 

result: torch.Size([2, 3, 10])
 tensor([[[2., 2., 2., 2., 2., 2., 2., 2., 2., 2.],
         [3., 3., 3., 3., 3., 3., 3., 3., 3., 3.],
         [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]],

        [[4., 4., 4., 4., 4., 4., 4., 4., 4., 4.],
         [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
         [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]]])
</code></pre>
<p>Question is, can I do this with some torch build-in operations?<br>
Thanks!</p>
What is the similar command as tf.flags.DEFINE_integer in pytorch

username: ZSheikhb
cooked: <p>I have been trying to convert a tensorflow code to pytorch. The code uses <code>tf.flags.DEFINE_integer</code>  etc to initialize different parameters of the model.</p>
<pre><code class="lang-auto">import tensorflow as tf
flags = tf.flags
flags.DEFINE_integer("batchSize", 20, "batch size.")
flags.DEFINE_integer("nEpochs", 1, "number of epochs to train.")
flags.DEFINE_float("adamLr", 1e-4, "AdaM learning rate.")
flags.DEFINE_integer("hidden_size", 200, "number of hidden units in en/decoder.")
flags.DEFINE_integer("latent_size", 50, "dimensionality of latent variables.")
flags.DEFINE_integer("K", 5, "number of components in mixture model.")
flags.DEFINE_string("experimentDir", "MNIST/", "directory to save training artifacts.")
inArgs = flags.FLAGS
</code></pre>
<p>What is the similar command-line as <code>tf.flags</code> in pytorch?</p>

username: ptrblck
cooked: <p>I don’t know how <code>tf.flags</code> are used, but assuming these values are used as arguments you could use plain Python variables:</p>
<pre><code class="lang-python">batch_size = 32
data = torch.randn(batch_size)
</code></pre>
<p>or in case it’s used to parse arguments you could use <code>argparse</code>.</p>
Plotting tensor

username: Nil_MSh
cooked: <p>Hi! How can I plot a tensor of size [1000][32][32]?</p>

username: ptrblck
cooked: <p>It depends how you would like to visualize it.<br>
You won’t be able to directly plot this tensor as e.g. an image, since <code>[1000, 32, 32]</code> doesn’t match an image format.<br>
However, you could create 1000 plots each visualizing a <code>32x32</code> grayscale image using <code>matplotlib</code>, but I don’t know if that’s what you are looking for.</p>
Reproducing the same hyper-parameter grid search results with every run

username: Vargabh
cooked: <p>Hi everyone,<br>
I am looking to perform grid search for all the hyper-parameters in any BNN method such MC-Dropout (drop-out rate, learning rate, weight decay for L2, no. of stochastic runs etc,…). I have fixed the random seed for e.g. (rng = np.random.RandomState(2)), but I am not able to reproduce the same results. Can anybody help me on how to reproduce the same results every-time I run my python code (uses python modules).</p>

username: ptrblck
cooked: <p>The <a href="https://pytorch.org/docs/stable/notes/randomness.html">Reproducibility docs</a> give you some information on how to get deterministic results.</p>
TensorFlow installation not found - running with reduced feature set

username: szahan
cooked: <p>Previously I had Tensorboardx 2.2 and all was good. But recently I had to reinstall Tensorboardx, first a later then downgraded to the same version but getting this warning and not all logging data is showing unless I use “–load_fast=false”. Also, have to reload to see new loggings. Any suggestions on how can I fix it? Thanks</p>
"Warning: NaN or Inf found in input tensor" but Input tensors do not contain NaN or Inf

username: ari9dam
cooked: <p>Hi,<br>
I am getting the following warning.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/7/73abfa63d7e7e7454bdbac6e7582e49a33b4d6da.png" data-download-href="https://discuss.pytorch.org/uploads/default/73abfa63d7e7e7454bdbac6e7582e49a33b4d6da" title="Capture"><img src="https://discuss.pytorch.org/uploads/default/original/2X/7/73abfa63d7e7e7454bdbac6e7582e49a33b4d6da.png" alt="Capture" data-base62-sha1="gvhmY0Yx5AgOcSHjetS708UCNKy" width="690" height="206" data-dominant-color="0C0D0C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture</span><span class="informations">1002×300 9.64 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I have checked that all the tensors that are either input to the model or being created inside the model are never NaN or Inf unlike this <a href="https://discuss.pytorch.org/t/warning-nan-or-inf-found-in-input-tensor/48628" class="inline-onebox">Warning: NaN or Inf found in input tensor</a>.</p>
<p>Could you please help me to resolve this issue. Thank you very much,</p>

username: ptrblck
cooked: <p>How did you make sure that no input value is a NaN or Inf?<br>
Could you share some code and maybe add an <code>assert</code> statement to raise an exception, if a NaN or Inf was detected?</p>

username: ari9dam
cooked: <p>I got the fix here <a href="https://github.com/lanpa/tensorboardX/pull/469" rel="nofollow noopener">https://github.com/lanpa/tensorboardX/pull/469</a></p>

username: szahan
cooked: <p>Hi, I am also getting the same error and have feeling might have something to do with tensorboard. Could you please share what did you changed. Your provided link is not quite clear</p>

username: RylanSchaeffer
cooked: <p><a class="mention" href="/u/szahan">@szahan</a> did you solve this issue? I have the exact same problem.</p>

username: szahan
cooked: <p>Hi, in my case it had nothing to do with tensorboard, instead, I was logging some values that had Nans in them, hence the error.</p>
PyTorch Profiler CUPTI warning

username: surya00060
cooked: <p>Hi!<br>
I have run into some CUPTI warning in PyTorch 1.9.0</p>
<p>In PyTorch 1.8.0,</p>
<pre><code class="lang-auto">with torch.autograd.profiler.profile(use_cuda=True) as prof:
 y = model(x)
prof.export_chrome_trace("trace.json")
</code></pre>
<p>The following code works and chrome trace shows both CPU and CUDA traces.</p>
<p>Whereas in PyTorch 1.9.0,</p>
<pre><code class="lang-auto">with torch.profiler.profile(activities=[torch.profiler.ProfilerActivity.CPU, torch.profiler.ProfilerActivity.CUDA]) as prof:
 y = model(x)
prof.export_chrome_trace("trace.json")
</code></pre>
<p>gives me a warning of <code>/home/surya/.local/lib/python3.6/site-packages/torch/autograd/profiler.py:440: UserWarning: CUPTI tracing is not available, falling back to legacy CUDA profiling   warn("CUPTI tracing is not available, falling back to legacy CUDA profiling") </code> and shows only CPU traces.</p>
<p>Any idea why this warning shows up and how to fix it? Thanks in advance for your reply.</p>
<p>I have been working on with Jetson Xavier AGX and installed PyTorch binaries from <a href="https://forums.developer.nvidia.com/t/pytorch-for-jetson-version-1-9-0-now-available/72048" rel="noopener nofollow ugc">here</a></p>

username: ptrblck
cooked: <p>I guess the linked binaries did not build PyTorch with CUPTI support so you could check, if you could rebuild them (or generally build from source).</p>

username: surya00060
cooked: <p>I rebuild the wheel from source by adding the flag <code>USE_KINETO=1</code>.<br>
Still facing the same issue.</p>
<pre><code class="lang-auto">&gt;&gt;&gt;torch.profiler.supported_activities()
{&lt;ProfilerActivity.CPU: 0&gt;}
</code></pre>
<p>Is there any flags which I have to setup for CUPTI support?<br>
Thanks in advance for any replies.</p>

username: ptrblck
cooked: <p>The build log should show if CUPTI was found and which support Kineto is thus using.<br>
Since the output only mentioned the CPU support I guess that CUPTI wasn’t found (or is not installed on your system).</p>

username: surya00060
cooked: <p>Hmm yeah. I checked up the logs and it showed CUPTI not found.</p>
<p>I was able to backtrack it and fix up the issue. Manually exporting <code>USE_CUPTI_SO=1</code> and rebuilding from source helped me fix the issue.</p>
<p>The default build option is setup with CUPTI off.</p>
<pre><code class="lang-auto">option(USE_KINETO "Use Kineto profiling library" ON)
option(USE_CUPTI_SO "Use CUPTI as a shared library" OFF)
</code></pre>
<p>Thanks again for your help.</p>

username: ptrblck
cooked: <p>Thank you for the follow up! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
What is a proper way to troubleshoot ddp speed gain?

username: sbelharbi
cooked: <p>hi,<br>
what is the proper way to troubleshoot the gain in speed with ddp ( <a href="https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel" rel="noopener nofollow ugc"><code>DistributedDataParallel</code></a>) in order to find bottlenecks?</p>
<p>i have 2 different models. model 1 (resnet) and model 2 (unet).<br>
size of model 2 (23million) &gt; size of model1 (32million)</p>
<p>in term of data, model2 does an additional disc access to load more sample-related data, while model1 only load the basic data.</p>
<p>we consider using either single gpu or 2 gpus.<br>
when using 2 gpus, model 1 gains 50% of speed compared to single gpu.<br>
in the other hand, model 2 only gain 15%.</p>
<p>i am using datalaoder with distributedsampler and 4 workers per process.<br>
every process uses only one gpu.</p>
<p>both gpus are in the same machine.<br>
use nccl backend.</p>
<p>thanks</p>

username: sbelharbi
cooked: <p>did some timing.<br>
using multi-gpus makes a type of loss acts up.<br>
the loss runs on cpu.<br>
so, not sure why is that.<br>
when using a single gpu, the loss takes 70ms to forward.<br>
when using multigpus, it takes 600ms.</p>
<p>looking why this is happening.<br>
the observed gain in time of 15% came from the validation that is done in multigpu.</p>
<p>thanks</p>

username: sbelharbi
cooked: <p>it turns out one of the losses is heavily using <a href="https://discuss.pytorch.org/t/how-multi-threading-is-managed-within-ddp/130896">multi-threading</a>…</p>

username: Yanli_Zhao
cooked: <p>I’m wondering why loss is not computed on GPUs?</p>

username: sbelharbi
cooked: <p>it is a custom c++ cpu-loss provided in another work.<br>
currently looking for a <a href="https://discuss.pytorch.org/t/c-cuda-custom-function-runtimeerror-cuda-error-invalid-device-function/131085">gpu</a> version.<br>
from initial run, gpu loss does not seem to be faster than the multi-threaded c++ cpu implementation.</p>
<p>still looking to find why and how to further speed up the gpu version.<br>
thanks</p>
How can I perform "Weighted NLLLoss" calculations manually?

username: andreys42
cooked: <p>Hello<br>
Im wonder why NLLLoss doesn’t change when using [2,2] weights in example below?<br>
There is really incomplete documentation related to details of calculations.</p>
<pre><code class="lang-auto">log_prob = torch.tensor([[0.1, 0.5],
                         [0.8, 0.8],
                         [0.8, 0.1]])

target = torch.tensor([0, 0, 1])

weight = torch.tensor([2.0, 2.0])

criterion = nn.NLLLoss()
criterion_weighted = nn.NLLLoss(weight=weight)

print('NLLLoss = {}'.format(criterion(log_prob, target)))
#NLLLoss = -0.3333333432674408

print('Weighted NLLLoss = {}'.format(criterion_weighted(log_prob, target)))
#Weighted NLLLoss = -0.3333333432674408

print('Manualy calculated weighted NLLLoss = {}'.format((-2*np.log(0.1) - 2*np.log(0.8) - 2*np.log(0.1)) / (3 * 4)))
Manualy calculated weighted NLLLoss = 0.8047189562170501

</code></pre>
<p>I guess there is some mistakes in my manual calculations. Any suggestion how can I calculate weighted NLLLoss manuall?</p>

username: KFrank
cooked: <p>Hi Андрей!</p>
<aside class="quote no-group" data-username="andreys42" data-post="1" data-topic="131353" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/andreys42/48/35546_2.png" class="avatar"> andreys42:</div>
<blockquote>
<pre><code class="lang-auto">print('Manualy calculated weighted NLLLoss = {}'.format((-2*np.log(0.1) - 2*np.log(0.8) - 2*np.log(0.1)) / (3 * 4)))
Manualy calculated weighted NLLLoss = 0.8047189562170501
</code></pre>
</blockquote>
</aside>
<p>You have two mistakes in your calculation:  First, <code>NLLLoss</code> does <em>not</em><br>
take the <code>log()</code> of its <code>input</code>, but you do in your manual calculation.<br>
Second, when you manually calculate the weighted average, you<br>
should be dividing by <code>3 * 2</code> (not <code>(3 * 4)</code>).  Thus:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import math
&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.9.0'
&gt;&gt;&gt; log_prob = torch.tensor([[0.1, 0.5],
...                          [0.8, 0.8],
...                          [0.8, 0.1]])
&gt;&gt;&gt; target = torch.tensor([0, 0, 1])
&gt;&gt;&gt; weight = torch.tensor([2.0, 2.0])
&gt;&gt;&gt; criterion = torch.nn.NLLLoss()
&gt;&gt;&gt; criterion_weighted = torch.nn.NLLLoss(weight=weight)
&gt;&gt;&gt; criterion (log_prob, target)
tensor(-0.3333)
&gt;&gt;&gt; criterion_weighted (log_prob, target)
tensor(-0.3333)
&gt;&gt;&gt; (-2 * math.log (0.1) - 2 * math.log (0.8) - 2 * math.log (0.1)) / (3 * 4)
0.8047189562170501
&gt;&gt;&gt; (-2 * (0.1) - 2 * (0.8) - 2 * (0.1)) / (3 * 2)
-0.3333333333333333
</code></pre>
<p>Best.</p>
<p>K. Frank</p>
Does DDP work on 2 cards with different number of GPUs?

username: BruceDai003
cooked: <p>Would it be possible to do DDP multi-nodes multi-gpus training if e.g., 1st card got 4 gpus, 2nd card got 4 gpus, and the 3rd got 2 gpus?</p>

username: Yanli_Zhao
cooked: <p>yes, it should work if you run one process on each GPU</p>
Object detection_yolov5s

username: Hasintha_Hewage
cooked: <h1>
<a name="i-want-to-run-a-trained-yolov5spt-in-a-local-pc-for-the-usage-of-an-opencv-application-how-can-i-do-that-in-code-as-a-new-to-pytorch-1" class="anchor" href="#i-want-to-run-a-trained-yolov5spt-in-a-local-pc-for-the-usage-of-an-opencv-application-how-can-i-do-that-in-code-as-a-new-to-pytorch-1"></a>I want to run a trained “yolov5s.pt” in a local pc for the usage of an OpenCV application. How can I do that in code  as a new to PyTorch</h1>
Saving/loading a dict using pytorch

username: Nil_MSh
cooked: <p>Hi! I have a code like this</p>
<pre><code class="lang-auto">calculate_kl(name):
kl = {'a': [], 'b': [], 'c': []}
        for i in range(64):
            value = calculate_value(name, a, i)    #returns an int
            kl['a'].append(value)
        for i in range(128):
            value = calculate_value(name, b, i)    # returns an int
            kl['b'].append(value)
        for i in range(256):
            value = claculate_value(name, c, i)    # returns an int
            kl['c'].append(value)
       return kl

importnace = {'alpha': calculate_kl('alpha'), 'beta': calculate_kl('beta')}
</code></pre>
<p>how can I save the dict importance and then use it in another code?<br>
I used <code>torch.save(importnace, 'kl.csv')</code> and then</p>
<pre><code class="lang-auto">importance = {}
importance.load_state_dict(torch.load('kl.csv'))
</code></pre>
<p>but it doesn’t work.</p>

username: huahuanZ
cooked: <p>You can load it with assignment</p>
<pre><code class="lang-auto">importance=torch.load(...) #importance should be OrderedDict type
</code></pre>
Changing value of variable while computing gradient

username: Akusa
cooked: <p>Hello,<br>
I would like to change the value of a variable while we compute the gradient between two iterations. As an example, consider the following code:</p>
<pre><code class="lang-auto">optimizer = torch.optim.Adam([{'params':param1, 'lr':1e-2}])
for ite in range(ite_max):
   optimizer.zero_grad()
   loss = compute_loss(param1)
   loss.backward()
   optimizer.step()

   # Here, we store the new value of the param1
  store_value = param1.detach().clone()

  # Now, I want to change the value of param1 to compute a new grad 
  # where the new_value change at each ite
  param1 = torch.tensor([new_value], requires_grad = True)
</code></pre>
<p>My question is about my last line because I guess it is wrong but I don’t really know how to do it.</p>