akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to create a custom data-set where the input is an image and a random number of robotic grasps

username: jsll
cooked: <p>Hi,</p>
<p>I’m working on an end-to-end robotic grasp planning method where the input to the network is an image, a grasp G (defined as a vector x,y,z) and an output v ranging from 0-1 that indicates how good the corresponding grasp is. The problem I’m facing is how to create a custom data-set to handle the case where one image have multiple possible grasps and outputs v.</p>
<p>For instance, when training the network, it should only take one image and one grasp at a time, but still be able to train on all or at least a subset of all grasps for that single image. If I, however, only load images as batches, I have no control on how to load the number of grasps corresponding to that image.</p>
<p>One option might be to create two separate data-sets, one for the image and one for the grasps and outputs, and load the data from these. I do, however,  hope for a more elegant solution where I only need one custom data-set.</p>

username: Balamurali_M
cooked: <p>You can form a list with this format. [(Img1, grasp1),(Img1, grasp2), (Img1, grasp3),…(Imgn, graspn)]. Iterate over this list using dataloader. If you need more information let me know. I can add a base code.</p>

username: jsll
cooked: <p>Thanks for your quick answer and your solution is a good idea. One downside, however, is that it requires multiple copies of the same image.</p>

username: Balamurali_M
cooked: <p>You can read the image during runtime.</p>

username: jsll
cooked: <p>So you mean I should only have the path to the image in  [(Img1, grasp1),(Img1, grasp2), (Img1, grasp3),…(Imgn, graspn)] and then when I actually need it I will read it in?</p>

username: Balamurali_M
cooked: <p>Perfect. Yeah. That’s how I will read 2d slices from 3d volumes. I don’t see a problem the same thing being adapted here.</p>
PyTorch 1.x Reinforcement Learning

username: Packt
cooked: <p><a href="https://www.packtpub.com/data/pytorch-1-0-reinforcement-learning-cookbook?utm_source=PyTorch&amp;utm_medium=forum&amp;utm_campaign=OutreachB13655" rel="nofollow noopener"># PyTorch 1.x Reinforcement Learning</a></p>
<p>Reinforcement learning (RL) is a branch of machine learning that has gained popularity in recent times. It allows you to train AI models that learn from their own actions and optimize their behavior. PyTorch has also emerged as the preferred tool for training RL models because of its efficiency and ease of use.</p>
<p>With this book, you’ll explore the important RL concepts and the implementation of algorithms in PyTorch 1.x. The recipes in the book, along with real-world examples, will help you master various RL techniques, such as dynamic programming, Monte Carlo simulations, temporal difference, and Q-learning. You’ll also gain insights into industry-specific applications of these techniques. Later chapters will guide you through solving problems such as the multi-armed bandit problem and the cartpole problem using the multi-armed bandit algorithm and function approximation. You’ll also learn how to use Deep Q-Networks to complete Atari games, along with how to effectively implement policy gradients. Finally, you’ll discover how RL techniques are applied to Blackjack, Gridworld environments, internet advertising, and the Flappy Bird game.</p>
<p>You’ll have developed the skills you need to implement popular RL algorithms and use RL techniques to solve real-world problems.</p>
Weights getting 'nan' during training

username: Shiv
cooked: <p>I am checking my weights every 10 epochs. I have tried xavier and normal initialization of weights and have varied learning rate in a wide range. Also, validation error either remain constant or increases slowly. Irrespective of various set up, I am getting ‘nan’ in some filters at 10th epoch. What could be the issue and how to solve it?</p>
<p>(1 ,0 ,.,.) =<br>
nan         nan         nan  …          nan         nan         nan<br>
nan         nan         nan  …          nan         nan         nan<br>
nan         nan         nan  …          nan         nan         nan<br>
…                   ⋱                   …<br>
nan         nan         nan  …          nan         nan         nan<br>
nan         nan         nan  …          nan         nan         nan<br>
nan         nan         nan  …          nan         nan         nan</p>

username: fmassa
cooked: <p>There can be several reasons.</p>
<ul>
<li>Make sure your inputs are not unitialized</li>
<li>check to see if you don’t have gradient explosion, that might lead to nan/inf. Smaller learning rate could help here</li>
<li>Check if you don’t have division by zero, etc</li>
</ul>
<p>It’s difficult to say more without further details.</p>

username: Shiv
cooked: <p><a class="mention" href="/u/fmassa">@fmassa</a> I have already tried very small learning rate but there is no affect.</p>

username: maxest
cooked: <p>I also had an issue with nan and it was caused by my error function (the one on which I backward) containing torch.sqrt (I wanted to measure my error in Euclidean distance). After removing that part (so now I use classical MSE) it all works without problems.<br>
I have to point out though that I had the very same code/algorithm implemented in TensorFlow, with sqrt included, and it worked without any nan problems.</p>

username: Dillon_Davis
cooked: <p>Were you using torch.sqrt on the output of MSELoss or did you remove torch.sqrt from somewhere in the source code of MSELoss? I’m having the same issue with MSELoss right now but I’m not explicitly using torch.sqrt myself.</p>

username: maxest
cooked: <p>I didn’t use MSELoss at all. I have my onw cost function in which I use (or not) troch.sqrt directly.</p>

username: damien
cooked: <p>Yes! After I decrease the learning rate, this problem dispear!</p>
In place padding and reindexing

username: Yonathan_Aflalo
cooked: <p>Hi.</p>
<p>I would like to know if there is a inplace version of the following series of operations on the tensor input: Basically, I have a tensor called input, and I would like to reshape and reindex along a specific dimension</p>
<p>The series of operations could be the following:</p>
<p>s=input.shape<br>
s[1] = new_len<br>
output = torch.zeros(s)<br>
output[:,index_tensor,…]=input</p>
<p>but the last one takes lots of time, almost as much as a convolution. Any idea how to get something similar but way faster?</p>
<p>Thanks</p>

username: ptrblck
cooked: <p>How large are the tensors and how did you define <code>index_tensor</code>?</p>
<p>I’m not sure, if your inplace assignment would work as you wish, since you could reassign some values, which would be needed in other steps:</p>
<pre><code class="lang-python"># unrolled
input[2] = input[1]
input[3] = input[2] # this values was reassigned in the previous step
</code></pre>
Does the model learn before or after transformations?

username: alx
cooked: <p>Hey I noticed my model learns a lot more when I’m passing big images although my transformations are the following:</p>
<pre><code class="lang-auto">train_trans = transforms.Compose([
    transforms.Resize(256),
    transforms.RandomCrop(_image_size),
    transforms.RandomHorizontalFlip(),
    transforms.ToTensor(),
    transforms.Normalize(_mean, _std),
])

</code></pre>
<p>When I feed 256x256 images directly, the model doesn’t learn much. This leads me to believe, something is learned prior to transformations. Can anyone confirm? Thanks</p>

username: anantguptadbl
cooked: <p><a class="mention" href="/u/alx">@alx</a><br>
The learning happens on variables ion the model. If there are transformations that have learnable Variables then it would lead to some learning</p>

username: Balamurali_M
cooked: <p>No. That can’t happen. The model learns better if the you have ample data which is given by data augmentations.</p>
BCELoss not accepting lists

username: anandaltekar
cooked: <p>Hi,</p>
<p>My convLSTM model returns a list of hidden states and my target is a list of 17 images( all  tensors)<br>
When the loss function is called, I get the following error:</p>
<blockquote>
<p>File “/Users/xyz/opt/anaconda3/envs/matrix/lib/python3.7/site-&gt;packages/torch/nn/modules/loss.py”, line 498, in forward<br>
return F.binary_cross_entropy(input, target, weight=self.weight, &gt;reduction=self.reduction)<br>
File “/Users/xyz/opt/anaconda3/envs/matrix/lib/python3.7/site-&gt;packages/torch/nn/functional.py”, line 2052, in binary_cross_entropy<br>
if target.size() != input.size():<br>
AttributeError: ‘list’ object has no attribute ‘size’</p>
</blockquote>
<p>Part of the training loop:</p>
<pre><code>hc = model.init_hidden(batch_size=1)
for batch_idx, (data, target) in enumerate(train_loader):
    optimizer.zero_grad()
    # Set target, images 2 to 18
    target = data[1:]
    if gpu:
        data = data.cuda()
        target = target.cuda()
        hc.cuda()
    # Get outputs of LSTM
    output = model(data, hc)
    # Calculate loss
    loss = criterion(output, target)
    loss.backward()
    optimizer.step()
</code></pre>
<p>How can I fix this?</p>

username: mailcorahul
cooked: <p>You need to pass torch.Tensor to criterion instead of list<br>
<code>target = torch.Tensor(data[1:])</code></p>

username: anandaltekar
cooked: <p>tried that. I’m still getting the same error. May be because ‘output’ is also a list of tensors? I tried casting output the same way. but I get a new error now:</p>
<blockquote>
<p>ValueError: only one element tensors can be converted to Python scalars</p>
</blockquote>

username: mailcorahul
cooked: <p>Can you post the full traceback?<br>
From the above traceback,  <code>target</code> seems to be the cause for exception.</p>

username: anandaltekar
cooked: <p>Here you go:</p>
<pre><code class="lang-auto">train.py:129: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  target = torch.tensor(data[1:])
Traceback (most recent call last):
  File "train.py", line 149, in &lt;module&gt;
    train(epoch)
  File "train.py", line 137, in train
    output = torch.tensor(output)
ValueError: only one element tensors can be converted to Python scalars</code></pre>

username: Balamurali_M
cooked: <p>Can you print shape and type of the variables data, target and hc ?</p>
Advice Loss function

username: anandaltekar
cooked: <p>What is a good loss function to calculate difference between a generated image and the ground truth. This is not a GAN. I’m trying to generate an image that is as close to the ground truth as possible. I’m using KLDivLoss in my model right now. Would like to hear more ideas though. Image size is (3,128,128)</p>

username: crowsonkb
cooked: <p>Typically people use MSELoss for that.</p>

username: anandaltekar
cooked: <p>Thanks. Just tried that.</p>

username: Balamurali_M
cooked: <p>If it is classification (segmentation), use cross entropy. If it is regression (detection, super-resolution), use mse,l1, smooth L1.</p>
Saving gradients after loss.backward()?

username: pytorchuser
cooked: <p>I’m attempting to save the gradients of some parameters with respect to my loss function. I want to take the values of the gradients, and use these values in another parameter in my network. However, it appears that these are not being retained (even with the retain_graph flag set to True). Is there any straightforward way to do this?</p>

username: albanD
cooked: <p>Hi,</p>
<p><code>retain_graph</code> has nothing to do with gradients.<br>
Could you give a small code sample that illustrate what you want to do please?</p>
The second parameter "tgt" of Transformer in its function forward()

username: FANTASTPATR
cooked: <p>I’m not very sure that what is the parameter “tgt” means.<br>
The document says it is the sequence of decoder. However, I have a question that since I have the sequence to decoder, thus I have the result of decoder, why do I have to use transformer for inferencing?In  another word, in inferencing process, we have no corresponded tgt to input.<br>
For example, I have a seq2seq task that translate “I love you” from English to Franch “Je t’aime”, what is the parameter of “tgt”?<br>
Looking forward answering. Thanks!</p>

username: xdwang0726
cooked: <p>tgt is the target input, in the training you need tgt input as golden truth to do teacher forcing learning, but in the inference, you don’t need tgt, you only need to input the encoded_vector from encoder and the first token in the tgt, usually it’s BOS.</p>
Why no Dropout in last layer of RNN

username: Wil_Helm
cooked: <p>What is the reason behind this restriction?</p>
<p>In the documentation for all recurrent layers is written:</p>
<p><strong>dropout</strong> – If non-zero, introduces a Dropout layer on the outputs of each RNN layer <strong>except the last layer</strong></p>
<p>But why? Is it an implementation issue? Or is there research on this topic?</p>
<p>When using only 1 LSTM layer I would not be able to use dropout, but it helps performance (when implemented manually) for my (time series forecasting) problem.</p>
<p>Thank you very much</p>

username: tom
cooked: <p>So the dropout in the last layer would be operating on what is the output of the RNN.<br>
This means you can do it yourself on the output if needed, an option you don’t have for the inner layers.<br>
Note that the dropout implemented by the RNN is not the dropout using one random draw for all timesteps.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Wil_Helm
cooked: <p>thanks that makes sense!</p>
How to calculate the big O complexity of an operation

username: Beinan_Wang
cooked: <p>Is there anyway we can calculate the big O complexity of an operation like topk() when executing on CPU and GPU respectively?</p>

username: LeviViana
cooked: <p>One easy way of checking out the complexity is just comparing the runtimes for proportional sizes of the input. For instance:</p>
<pre><code class="lang-bash">python -m timeit --setup="import torch;x = torch.rand(10 ** 5)" "x.topk(10)" 
python -m timeit --setup="import torch;x = torch.rand(10 ** 6)" "x.topk(10)" 
</code></pre>
<p>For the cuda implementation we get:</p>
<pre><code class="lang-bash">python -m timeit --setup="import torch;x = torch.rand(10 ** 5).cuda()" "x.topk(10);torch.cuda.synchronize()" 
python -m timeit --setup="import torch;x = torch.rand(10 ** 6).cuda()" "x.topk(10);torch.cuda.synchronize()" 
</code></pre>
<p>I ran these tests several times on my machines, everything looks linear (with respect to the size of the vector).</p>
What is the difference between rank and local-rank?

username: AlexLuya
cooked: <p>The doc confuses me quite a lot,would you please tell me:</p>
<ol>
<li>What is the difference of two?</li>
<li>When must use rank and when must use local-rank?</li>
</ol>

username: spanev
cooked: <p>Hi <a class="mention" href="/u/alexluya">@AlexLuya</a>,</p>
<p>In the context of multi-node training, you have:</p>
<ul>
<li>
<code>local_rank</code>, the rank of the process on the local machine.</li>
<li>
<code>rank</code>, the rank of the process in the network.</li>
</ul>
<p>To illustrate that, let;s say you have 2 nodes (machines) with 2 GPU each, you will have a total of 4 processes (p1…p4):</p>
<pre><code class="lang-auto">            |    Node1  |   Node2    |
____________| p1 |  p2  |  p3  |  p4 |
local_rank  | 0  |   1  |  0   |   1 |
rank        | 0  |   1  |  2   |   4 |
</code></pre>

username: AlexLuya
cooked: <p><a class="mention" href="/u/spanev">@spanev</a>,thanks,if p3 wants to send sth to p4,<br>
1,it can use either local_rank or rank<br>
2,but for performance,it should use local_rank<br>
Am I right about above two?<br>
3,why not just all use rank,and let lib to decide to which seed method(cross process or cross node) to use,what is the case that developer must use local_rank?</p>

username: pritamdamania87
cooked: <p>I’m assuming you’re referring to <code>local_rank</code> mentioned here: <a href="https://github.com/pytorch/pytorch/blob/master/torch/distributed/launch.py" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/torch/distributed/launch.py</a></p>
<ol>
<li>You should use <code>rank</code> and not <code>local_rank</code> when using torch.distributed primitives (send/recv etc). local_rank is passed to the training script only to indicate which GPU device the training script is supposed to use.</li>
<li>You should always use <code>rank</code>.</li>
<li>
<code>local_rank</code> is supplied to the developer to indicate that a particular instance of the training script should use the “local_rank” GPU device. For illustration, in the example above provided by <a class="mention" href="/u/spanev">@spanev</a>, p1 is passed local_rank 0 indicating it should use GPU device id 0.</li>
</ol>
Why the c++ frontend cost so much memory?

username: abtion
cooked: <p>Hi, I wrote a function to encode the text Ids by albert，but it cost much more memory than python. for 1.6k sentences, will cost more than 60GB memory, but every sentences are less than 10 words. Here is my code:</p>
<pre><code class="lang-auto">torch::Tensor encode(torch::Tensor texts, torch::Tensor attention_masks, torch::jit::script::Module model) {
    model.eval();
    assert(texts.size(0) == attention_masks.size(0) &amp;&amp; texts.size(1) == attention_masks.size(1));

    int n = texts.size(0);
    int _batch_size = 128;
    int nums = n / _batch_size;
    if (n % _batch_size != 0) {
        ++nums;
    }
    std::vector&lt;torch::jit::IValue&gt; inputs_texts;
    std::vector&lt;torch::jit::IValue&gt; inputs_attention_masks;
    for (int i = 0; i &lt; nums; i++) {
        int start = i * _batch_size;
        int end = start + _batch_size;
        inputs_texts.push_back(texts.slice(0, start, end));
        inputs_attention_masks.push_back(attention_masks.slice(0, start, end));
    }
    torch::Tensor rst;
    for (int i = 0; i &lt; nums; i++) {
        std::vector&lt;torch::jit::IValue&gt; inputs;
        inputs.push_back(inputs_texts[i]);
        inputs.push_back(inputs_attention_masks[i]);
        torch::Tensor temp_rst = model.forward(inputs).toTensor();

        if (i == 0) {
            rst = temp_rst;
        } else {
            rst = torch::cat({rst, temp_rst}, 1);
        }

    }

    return rst;
}

</code></pre>
<p>While the model is a torchscript model of albert small, I have modifid it to output the hidden_state.</p>

username: albanD
cooked: <p>Given that you’re doing only inference, you can add at the beginning of your function <code>NoGradGuard guard;</code>. This will disable the autograd and any extra buffer.<br>
Does that help?</p>
Implicit asymmetric padding in Conv2d?

username: agaldran
cooked: <p>Hi,</p>
<p>I have been banging my head against this for one hour already. In the original Resnet paper they say that the first layer of all residual networks starts with a convolutional layer with filters of size 7x7 and stride=2. In order to make that work, in <code>resnet.py</code> from <code>torchvision</code> this is implemented with a padding of 3 pixels:</p>
<pre><code class="lang-auto">self.conv1 = nn.Conv2d(3, self.inplanes, kernel_size=7, 
                                      stride=2, padding=3, bias=False)
</code></pre>
<p>Now, according to the standard formula for input/output in conv2d layers, if the input is the standard 224x224 imagenet size, one should get out of the above operation an output of size (224-7+2x3)/2. The point is that in this formula the numerator is uneven. So how is it that this does not even crash?</p>
<p>Summarizing, here is the question. Why does this even work:</p>
<pre><code class="lang-auto">import torch

conv1 = torch.nn.Conv2d(in_channels=3, out_channels=3, kernel_size=7, 
                                         stride=2, padding=3)

tensor = torch.ones([1,3,224,224])

print(conv1(tensor).shape)
</code></pre>
<p>and returns a tensor of spatial size 112x112? Is <code>Conv2d</code> performing some kind of padding but only to the right and bottom of the input volume?</p>
<p>Many thanks!</p>

username: Pedro_Costa
cooked: <p>Hi,</p>
<p>In those situations Conv2d discards the last columns/rows of the input, after padding, as mentioned in PyTorch documentation: <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.Conv2d" rel="nofollow noopener">Conv2d</a>.</p>
<p>In the particular case you are mentioning, that is the same as having a right/bottom padding of 2 instead of 3, as the last column/row is discarded. Translating to the formula that you mentioned to compute the output dimension of the convolution we get (224-7 + 3+2) / 2  + 1 = 112.</p>

username: agaldran
cooked: <p>That makes sense, thank you!</p>
Install PyTorch without MKL-DNN support using pip?

username: akrisilias21
cooked: <p>Is this possible somehow? Or at least to disable MKL-DNN on runtime?</p>
<p>I found this for runtime:<br>
torch.backends.mkldnn.flags(enabled=False)<br>
but I get the following error:<br>
Traceback (most recent call last):<br>
File “”, line 1, in <br>
AttributeError: module ‘torch.backends’ has no attribute ‘mkldnn’</p>
Deploy Pytorch model under Android with Python via BeeWare

username: Chame_call
cooked: <p>Can I deploy Pytorch model under Android with Python via BeeWare?</p>

username: Chame_call
cooked: <p>Is that possible at least?</p>

username: tom
cooked: <p>I don’t think it would be what you expect.<br>
Currently (and as far as I know), PyTorch doesn’t build the Python parts for Android.<br>
It would probably be possible to have a thin wrapper around the JIT parts (similar to what the Java offering), but it’s not been done.</p>
<p>Best regards</p>
<p>Thomas</p>
About nn.Linear(16*5*5,120)

username: cclover
cooked: <p>Hi, I’m just getting started with the pytorch.Preformatted text<br>
In this example, I don’t understand why the input of the full connected  layer is the output latitude of the previous convolution multiplied by the size of the convolution kernel?<br>
This is my code</p>
<pre><code class="lang-auto">class net(nn.Module):
    def __init__(self):
        super(net,self).__init__()
        self.cov1 = nn.Conv2d(1,6,5)#1表示图像为单通道  6表示输出的时候是6个通道  5表示卷积核5*5
        self.cov2 = nn.Conv2d(6,16,5)
        #接下来是仿射层/全连接层
        #nn.Linear的第一个参数取决于上一层的输出，上一个Cov2d输出纬度是16，然后还要乘以卷积核的长和宽
        self.fc1 = nn.Linear(16*5*5,120)
        self.fc2 = nn.Linear(120,81)
        self.fc3 = nn.Linear(81,10)
</code></pre>

username: ptrblck
cooked: <p>The <code>5*5</code> are not derived from the kernel size, but the spatial size of the input to the linear layer, which is apparently also <code>5*5</code>.<br>
You can add a print statement right before feeding the activation to <code>self.fc1</code> to see the activation shape, which should be <code>[batch_size, 16, 5, 5]</code> before flattening.</p>

username: cclover
cooked: <p>thank you very much！！！</p>

username: cclover
cooked: <aside class="quote quote-modified" data-post="10" data-topic="14644">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/abdulsalambande/48/40450_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/input-size-of-fc-layer-in-tutorial/14644/10">Input size of fc layer in tutorial?</a> <a class="badge-category__wrapper " href="/c/vision/5"><span data-category-id="5" data-drop-close="true" class="badge-category " title="Topics related to either pytorch/vision or vision research related topics"><span class="badge-category__name">vision</span></span></a>
  </div>
  <blockquote>
    The input of a Pytorch Neural Network is of type  [BATCH_SIZE] * [CHANNEL_NUMBER] * [HEIGHT] * [WIDTH]. 
Example  : So lets assume you image is of dimension 1×3×32×32 meaning that you have 1 image with 3 channels (RGB) with height 32 and width 32. So using the formular of convolution which is  ((W - F + 2P)/ S )+1 
and  ((H - F + 2P)/ S )+1 . The first one is for the  Width and the second one is for the height 
NOTE : Delete the the 5th line of the code because you already have pooling in the 12…
  </blockquote>
</aside>

<p>This might help</p>
Converting Keras to Pytorch

username: 111191
cooked: <p>Blow is my keras  and torch code</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/1/01410ccc4cc8f26315eca2f7d910447e8e8b0bfc.png" data-download-href="https://discuss.pytorch.org/uploads/default/01410ccc4cc8f26315eca2f7d910447e8e8b0bfc" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/1/01410ccc4cc8f26315eca2f7d910447e8e8b0bfc_2_690x323.png" alt="image" data-base62-sha1="b5QtHtJMR9z7cg4UoJ2KQnHzhW" width="690" height="323" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/1/01410ccc4cc8f26315eca2f7d910447e8e8b0bfc_2_690x323.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/1/01410ccc4cc8f26315eca2f7d910447e8e8b0bfc_2_1035x484.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/1/01410ccc4cc8f26315eca2f7d910447e8e8b0bfc_2_1380x646.png 2x" data-dominant-color="F3F4F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1675×786 227 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Q :<br>
I am wondering why the performance is poor when implementing with torch.</p>

username: ptrblck
cooked: <p><code>nn.CrossEntropyLoss</code> expects raw logits, so you should remove the softmax at the end of your model.</p>

username: 111191
cooked: <p>Thanks you very much!</p>
Confusion about torch.nn.Transformer

username: LiHaibo
cooked: <p>hi, I’m a bit confusing with src_mask and src_key_padding_mask, the explanation on pytorch docs are<br>
<strong>src_mask</strong>  – the additive mask for the src sequence (optional).<br>
<strong>src_key_padding_mask</strong>  – the ByteTensor mask for src keys per batch (optional).<br>
In my opinion, src_mask 's dimension is (S,S), and S is the max source length in batch, so i need to send input src_mask (N,S,S) to the Transformer.I don’t know if i understand that correctly. I don’t understand the src_key_padding_mask’s explanation on website docs, this is confusing me.<br>
for the provided example code ,<br>
output = transformer_model(src, tgt, src_mask=src_mask, tgt_mask=tgt_mask)<br>
set the [src/tgt/memory]_key_padding_mask are None as default, I’m a little confused about this operation.</p>
ERROR unexpected indent

username: JaeGer
cooked: <p>hello ,<br>
I have this error and I don’t knwo how to fix . Thank you.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/b/b307a205f10537886144ab26d78970b5372906a7.png" data-download-href="https://discuss.pytorch.org/uploads/default/b307a205f10537886144ab26d78970b5372906a7" title="Screenshot%20from%202018-05-24%2013-00-27"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/b/b307a205f10537886144ab26d78970b5372906a7_2_690x247.png" alt="Screenshot%20from%202018-05-24%2013-00-27" data-base62-sha1="pxLOSv6iZTU8izE03NcaPNM5zM3" width="690" height="247" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/b/b307a205f10537886144ab26d78970b5372906a7_2_690x247.png, https://discuss.pytorch.org/uploads/default/optimized/2X/b/b307a205f10537886144ab26d78970b5372906a7_2_1035x370.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/b/b307a205f10537886144ab26d78970b5372906a7_2_1380x494.png 2x" data-dominant-color="FAFAFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot%20from%202018-05-24%2013-00-27</span><span class="informations">1853×665 150 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ReLuTropy
cooked: <p>are there some space before self.fc2 ?</p>

username: lkywk
cooked: <p>there is a space before self.fc2. Make them in a line.</p>

username: creigelde
cooked: <p>As the error message indicates, you have an <a href="http://net-informations.com/python/err/indent.htm" rel="nofollow noopener">unexpected indent</a> error. This error occurs when a statement is unnecessarily indented or its indentation does not match the indentation of former statements in the same block. Python not only insists on indentation, it insists on consistent indentation . You are free to choose the number of spaces of indentation to use, but you then need to stick with it. If you indent one line by 4 spaces, but then indent the next by 2 (or 5, or 10, or …), you’ll get this error. Whenever you have a situation with code inside of a statement, that inside code must be indented, and must be indented consistently. It is used by the interpreter to know how to delimit blocks of instructions.</p>
Box intersection-over-union returns zero

username: elbek
cooked: <p>Not sure why following two boxes return <code>0</code> for <code>box_iou</code> value</p>
<pre><code class="lang-auto">import torchvision.ops as ops
b1 = torch.tensor([167, 238, 249, 276]).view(1, -1)
b2 = torch.tensor([146, 230, 228, 268]).view(1, -1)
print(ops.boxes.box_iou(b1, b2))
</code></pre>
<p>They look like nicely overlapped.</p>
<p>What I am missing?</p>
<p>Thanks in advance</p>

username: elbek
cooked: <p>ahh, b1 and b2 had to be float.<br>
making this gives right value:</p>
<pre><code class="lang-auto">b1 = torch.tensor([167, 238, 249, 276]).view(1, -1) * 1.0
b2 = torch.tensor([146, 230, 228, 268]).view(1, -1) * 1.0
</code></pre>
About torchvision for C++ frontend

username: chenglu
cooked: <p>The C++ frontend of PyTorch seems not completed to me. If we want to train a model in Python and deploy that model in C++, we have to do hard about code porting. And I think most likely is the part of data preprocessing which could be implemented by torchvision in Python, but it’s missing in C++.</p>
<p>The <code>torchvision</code> package in Python mainly have 2 modules:</p>
<ul>
<li>torchvision.models</li>
<li>torchvision.transforms</li>
</ul>
<p>For C++ to use these,<br>
The <code>torchvision.models</code> I think may be easily ported by JIT. The painful part is the <code>torchvision.transform</code>. If I used the code from the module in training phase, does it mean that I have to implement the same transforms in C++ now? Is the torchvision for C++ in schedule?</p>

username: tom
cooked: <p>Indeed, torchvision C++ support isn’t matching Python support.<br>
However, <a class="mention" href="/u/shahriarss">@ShahriarSS</a> is doing some good work on it, so the gap is getting smaller.</p>
<p>My guess is that most people use OpenCV to do transforms or do them manually. (Personally, I incorporated things like “normalizing” into the traced/scripted model last time I did this.)</p>
<p>Best regards</p>
<p>Thomas</p>

username: ShahriarSS
cooked: <p>Hi Thomas. The progress in TorchVision’s C++ side: Models have been written and I will start working on Transforms soon.</p>

username: chenglu
cooked: <p>Hi sshahriarSS, is there a github repo that you working on?</p>

username: ShahriarSS
cooked: <p>The models exist in TorchVision repository which can be installed using cmake.</p>

username: chenglu
cooked: <p>Great, saw it already on the master branch <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>

username: erict
cooked: <p>Hi <a class="mention" href="/u/shahriarss">@ShahriarSS</a>,</p>
<p>thanks for your work. Any update for Transforms in C++?</p>

username: ShahriarSS
cooked: <p>Hi <a class="mention" href="/u/erict">@erict</a>. It has been a while since I worked on torchvision and I have some incomplete PRs there. Unfortunately I don’t have the time to work on Transforms or other stuff right now.</p>

username: erict
cooked: <p>Hi <a class="mention" href="/u/shahriarss">@ShahriarSS</a>,</p>
<p>ok i will try to do something by myself based on your work. If i get something useful i will share here. Have you some hints on which could be the best way? for example i’m planning to use opencv to do some transform. Do you think could be a good starting point?</p>

username: ShahriarSS
cooked: <p><a class="mention" href="/u/erict">@erict</a>  I suggest you open an issue in torchvision and discuss the design there and then start. This way you will know how to start and it’ll be approved by torchvision. Feel free to mention me so I can help.</p>
Calculating gradient of input except the gradient of model parameter

username: Juna
cooked: <p>Hello,<br>
I’m trying to get the gradient of input but without calculating the gradient of model parameters.<br>
More specifically, there is an input A and this goes into a model M and then output P comes out.<br>
With this output P and the input A, I will make another input B (for example, elementwise multiplication bla bla) and this goes into the model M and then I get final output Q.<br>
Now, the loss function is gonna be formulated with this output Q and given label.</p>
<p>To make things clear, I will do<br>
input A -&gt; model M -&gt; output P (process <span class="hashtag">#1</span>)<br>
C(a result of doing something with an output P and input A) -&gt; model M -&gt; output Q  (process <span class="hashtag">#2</span>)<br>
and then I will do</p>
<p>loss = criterion(output Q, label)<br>
loss.backward()</p>
<p>but the point is</p>
<ol>
<li>the gradient of loss wrt the model M’s parameter in process <span class="hashtag">#2</span><br>
should not be saved in model.something.weight.grad and model.something.bias.grad</li>
<li>but the gradient of loss wrt C in process <span class="hashtag">#2</span> should be calculated and saved in C.grad.</li>
</ol>
<p>Backpropagation flows through C and goes to process <span class="hashtag">#1</span>.<br>
Here in process <span class="hashtag">#1</span>, on the other hand,</p>
<ol start="3">
<li>the gradient of loss wrt the model M’s parameter should be calculated and saved.</li>
</ol>
<p>As far as I know, .detach() makes all the superior parameters(parameters that are closer to the input) not calculate their gradient.<br>
Is there any way to do this?</p>

username: albanD
cooked: <p>Hi,</p>
<p>The problem is that M is the same during process 1 and 2. You won’t be able to get 2 different behaviors here.</p>
<p>If would suggest doing two steps (add the corresponding optimizer zero grad and step of course):</p>
<pre><code class="lang-python"># Process 1 as usual:
P = M(A)
C = some_func(P)

# Now break the graph and get a new C
C_2 = C.detach().requires_grad_()

# Process 2
Q = M(C_2)
loss = criterion(Q, label)

# Compute grads for C_2 (but don't change .grads in M)
C_2_grad = autograd.grad(loss, C_2)[0]

# Not do a regular backward for process 1 to populate the .grad in M
C.backward(C_2_grad)

</code></pre>

username: Juna
cooked: <p>Thanks a lot. It works!!</p>
DistributedDataParallel gradient print

username: sherdencooper
cooked: <p>Hi all! New to pytorch and i am using pytorch to do distributed training. I knew that ‘DistributedDataParallel’ averages gradients between each process. I want to know that whether i could print the gradient before average for every process ?</p>

username: pietern
cooked: <p>Hi! This is possible. DDP relies on <code>torch.autograd.backward</code> to accumulate the gradients into the <code>grad</code> tensor of the model parameters. There is a functional alternative in <code>torch.autograd.grad</code> that doesn’t accumulate at all. If you’re interested in the local gradients, instead of running <code>loss.backward()</code>, you can run <code>torch.autograd.grad(loss, model.parameters())</code> and get back a list of gradient tensors, one for every model parameter. This doesn’t accumulate them into the <code>grad</code> tensor of the model parameter, so it doesn’t kick off DDP. If you want to run DDP afterwards anyway, make sure to pass the <code>retain_graph=True</code> kwarg to <code>torch.autograd.grad</code>. I haven’t tried any of this out, but in theory it should work.</p>

username: sherdencooper
cooked: <p>Thanks a lot ! Your advice is really helpful ! I got the local gradients and it seems that DDP is not affected at all ! Its very cool that i can get professional guidance at pytorch’s forum.</p>

username: pietern
cooked: <p>I’m glad you were able to continue! <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>

username: meilu_zhu
cooked: <p>Hi, pietern <a class="mention" href="/u/pietern">@pietern</a> , the “DistributedDataParallel” automatically average the gradient when calling “loss.backward()”,<br>
But I didn’t find the corresponding script about how to get all the gradient of nodes and average the gradients during backward in pytorch source code, Do you know where it is ?</p>
What's this code mean?

username: a_new_pytorcher
cooked: <p>[1&gt;&gt;d &amp; 1 for d in range(10)][::-1]</p>

username: Eta_C
cooked: <p>Just some simple bit operations, see <a href="https://wiki.python.org/moin/BitwiseOperators" rel="nofollow noopener">here</a></p>
<pre><code class="lang-python">[d for d in range(10)] -&gt; [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
[1 &gt;&gt; d for d in range(10)] -&gt; [1, 0, 0, 0, 0, 0, 0, 0, 0, 0]
[1 &gt;&gt; d &amp; 1 for d in range(10)] -&gt; [1, 0, 0, 0, 0, 0, 0, 0, 0, 0]
[1 &gt;&gt; d &amp; 1 for d in range(10)][::-1] -&gt; [0, 0, 0, 0, 0, 0, 0, 0, 0, 1]
</code></pre>

username: a_new_pytorcher
cooked: <p>thank you,i am a green hand,and dont know a lot about it .</p>