akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Question about Index_selct in 4d-tensor

lxtGH: <p>Suppose we have a tensor <strong>T</strong> with size(b, h * w, d) and an  tensor <strong>Index</strong> with size(b, h * w, k)-&gt; with range(0, h*w-1) which stores the k different indexes of tensor <strong>T</strong> for each value of b * h * w.</p>
<p>How to generate indexed tensor from T with size (b, h * w, k, d) <strong>without extra memory cost</strong> ?<br>
A memory cost way to implement this is:</p>
<pre><code class="lang-auto">T= T.unsqueeze(1).expand(b,h*w,h*w,d)
Index = I.unsqueeze(2).expand(b,h*w,k,d)
res =torch.gather(T,2,Index)
</code></pre>
In C++ API how to get the function type in a Functional module?

nkg114mc: <p>Hi all,</p>
<p>I am a beginner user of PyTorch. As mentioned in the title, my question is: how can I  get the function type in Functional module in C++ API? For example, if there are two activation modules in my network:</p>
<p>torch::nn::Functional activ1(torch::relu)<br>
torch::nn::Functional activ2(torch::tanh)</p>
<p>Then I pass each of them to another procedure, say void print_name(torch::nn::Module &amp;m) which print the name of current module, then how can I know that activ1 is ReLU and activ2 is Tanh? I try to call the name() but both of them only return “torch::nn::Functional”…</p>
<p>BTW: Is there any way that I can override their module name? If this works then it can also indirectly resolve my question.</p>
<p>Thanks in advance!</p>

nkg114mc: <p>Hi all,</p>
<p>I found a stupid but workable solution for this: just write two new activation module classes that inherit torch::nn::FunctionalImpl, and then override their name_ in the constructor. The code is like following:</p>
<pre><code class="lang-auto">struct ActivationTanhImpl : torch::nn::FunctionalImpl {
public:
	ActivationTanhImpl() :
		torch::nn::FunctionalImpl(torch::tanh),
		torch::nn::Module("torch::nn::TanhImpl") {}
};
TORCH_MODULE(ActivationTanh);

struct ActivationReLUImpl : torch::nn::FunctionalImpl {
public:
	ActivationReLUImpl() :
		torch::nn::FunctionalImpl(torch::relu),
		torch::nn::Module("torch::nn::ReLUImpl") {}
};
TORCH_MODULE(ActivationReLU);
</code></pre>
<p>I know this is not elegant solution, but currently that’s the only workable one I can come out. Please let me know if you have any suggestions for this question. Thanks again!</p>
Expected object of backend CPU but got backend CUDA for argument

Donggeun: <p>I used .to (device) to move both model and data to gpu. However, an error occurs.</p>
<p>My Source: <a href="https://github.com/yudonggeun/text2speech" rel="nofollow noopener">https://github.com/yudonggeun/text2speech</a></p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/home/modeep/Documents/GitHub/Text2Speech/train.py", line 493, in &lt;module&gt;
    main()
  File "/home/modeep/Documents/GitHub/Text2Speech/train.py", line 490, in main
    train_init(config.model_dir, config, multi_speaker)
  File "/home/modeep/Documents/GitHub/Text2Speech/train.py", line 285, in train_init
    config=config, multi_speaker=multi_speaker)
  File "/home/modeep/Documents/GitHub/Text2Speech/train.py", line 368, in train
    y_pred = model(multi_speaker, inputs, sorted_lengths, loss_coeff, mel_targets, linear_targets, stop_token_target, speaker_id)
  File "/home/modeep/anaconda3/envs/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/modeep/anaconda3/envs/conda/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 152, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/home/modeep/anaconda3/envs/conda/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 162, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/home/modeep/anaconda3/envs/conda/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 83, in parallel_apply
    raise output
  File "/home/modeep/anaconda3/envs/conda/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 59, in _worker
    output = module(*input, **kwargs)
  File "/home/modeep/anaconda3/envs/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/modeep/Documents/GitHub/Text2Speech/tacotron/tacotron.py", line 39, in forward
    char_embedded_inputs = F.embedding(inputs, self.char_embed_table)
  File "/home/modeep/anaconda3/envs/conda/lib/python3.6/site-packages/torch/nn/functional.py", line 1506, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: Expected object of backend CPU but got backend CUDA for argument #3 'index'
</code></pre>

alex.veuthey: <p>I’m not sure this is the cause, but you also want to call <code>.to(device)</code> on your criterion!</p>

Donggeun: <p>I want to use multi gpu.</p>
<p>So how do i change the .to (device)?</p>

alex.veuthey: <p>The way you did it is fine. Just add the <code>.to(device)</code> on your criterion, I don’t think you need to use <code>DataParallel</code> for it.</p>

Donggeun: <p>I delete and run. but same error occurs.</p>
<p>Delete:</p>
<pre><code class="lang-auto">    if torch.cuda.device_count() &gt; 1:
        print("Use", torch.cuda.device_count(), "GPUs")
        model = nn.DataParallel(model)
</code></pre>
Using torch.nn.DataParallel with variable-sized input tensors

Feynman27: <p>I have a (faster r-cnn) model that currently supports a variable-sized input tensor of size (1,3,H,W) with an image batch size of 1. I’d like to run this model on two GPUs, using 1 image per GPU.</p>
<p>However, I can’t figure out how to send 2 variable-sized input tensors to each gpu. The example  <a href="http://pytorch.org/docs/master/nn.html#dataparallel-layers-multi-gpu-distributed" rel="nofollow noopener">here</a> assumes that the variable <code>input_var</code> is a concatenation of the inputs along the batch dimension:</p>
<pre><code class="lang-python">&gt;&gt;&gt; net = torch.nn.DataParallel(model, device_ids=[0, 1, 2])
&gt;&gt;&gt; output = net(input_var)
</code></pre>
<p>How would I distribute a model of image batch size 2 and variable input H and W across 2 gpus?</p>

richard: <p>Interesting. I think DataParallel assumes same sizes. One thing you could do is pad both images to some larger size and modify the inputs to specify H and W.</p>

Feynman27: <p>One of my colleagues suggested the same thing. Appears that’s the best option at the moment.</p>

gollum: <p>I have a similar kind of problem with my model. To deal with variable size input I’ve implemented a custom collate function. Similar to <a href="https://github.com/txie-93/cgcnn/blob/b98c1cd109da971f599dd492ac7a5dad2c29e886/cgcnn/data.py#L96" rel="nofollow noopener">this</a><br>
I want to parallelize my model too.<br>
Did this work for you?? It would be helpful if you can explain in a bit detail. Thankyou so much in advance.</p>
Global max pooling?

Fangzou_Liao: <p>Hi, I am looking for the global max pooling layer.</p>
<p>Because in my case, the input shape is uncertain and I want to use global max pooling to make their shape consistent. But I do not find this feature in pytorch?</p>

fmassa: <p>You can use the functional interface of max pooling for that. In you forward function:</p>
<pre><code class="lang-python">import torch.nn.functional as F
output = F.max_pool2d(input, kernel_size=input.size()[2:])
</code></pre>

Ilya_Ezepov: <p>You can do something simpler like</p>
<pre><code class="lang-auto">import torch
output, _  = torch.max(input, 1)
</code></pre>
<p>(Be careful with the second argument. For 2d pooling it should be something like <code>(1, 2)</code> )</p>
How to pass non-tensor type elements from one process to another with torch distributed

hanbit: <p>Hi,</p>
<p>I’m currently using torch.distributed and want to pass non-tensor type (e.g. dictionary, numpy etc.) elements from one process to another. I have considered making non-tensor type to tensor type before, but the problem is that each element has a different size so that it is hard to gather them to one local process. Do you have any idea?</p>

hanbit: <p>I’ve got an idea from <a href="https://github.com/facebookresearch/maskrcnn-benchmark" rel="nofollow noopener">the repository</a> of official Mask RCNN. They cast any picklable object into byteTensor and gather it from each rank.</p>
MLIR - a common intermediate representation (IR)

PeterCDMcLean: <p>MLIR: <a href="https://github.com/tensorflow/mlir" rel="nofollow noopener">https://github.com/tensorflow/mlir</a></p>
<p>MLIR’s intention seems to be an IR lowering framework. In my opinion, this has great synergy with the multiple levels of IR that Glow currently provides.</p>
<p>Does Glow have any intention / interest of integration or use-of MLIR?</p>

jfix: <p>Hi Peter, we don’t have any plans for MLIR for now. It could make sense to load MLIR into Glow (converting MLIR into Glow IR), which would allow us to use Glow’s optimization stack, and target any of our backends. Did you have something in particular in mind for Glow + MLIR?</p>
Glow master branch ninja test

Andre_Chang: <p>Hi,<br>
I git cloned <a href="https://github.com/pytorch/glow" rel="nofollow noopener">https://github.com/pytorch/glow</a><br>
and followed the build instructions. Tried with both release and debug build mode</p>
<p>When I ran ninja test, it gave segfault for the tests.</p>
<p>I am running on Ubuntu18.04.02</p>
<p>I am missing something?</p>
<pre><code class="lang-auto">The following tests FAILED:
	  1 - BackendCorrectnessTest (SEGFAULT)
	  2 - BackendTest (SEGFAULT)
	  6 - DeviceManagerTest (SEGFAULT)
	 11 - GradCheckTest (SEGFAULT)
	 17 - HyphenTest (SEGFAULT)
	 21 - MLTest (SEGFAULT)
	 25 - OperatorTest (SEGFAULT)
	 28 - QuantizationTest (SEGFAULT)
	 29 - TensorsTest (SEGFAULT)
	 31 - TraceEventsTest (SEGFAULT)
</code></pre>

jfix: <p>Hi Andre, no idea what’s going wrong here without additional details. Can you run one of these tests, e.g. OperatorTest, perhaps with a debugger, and give more details, like a backtrace?</p>
Tensor identities not working. Bug or feature?

Florian_Dietz: <p>I want to compare two tensors by identity / by reference. Normally, comparing two objects in python by identity works with ‘is’ or by comparing id() values. However, this does not seem to work:</p>
<p>a = torch.tensor([1,2,3])<br>
b = a.detach()<br>
print(a.data is b.data) # False<br>
print(id(a.data) == id(b.data)) # True</p>
<p>Shouldn’t both of these statements be True?<br>
It looks like I can make a reference comparison with id()==id(), but I am reluctant to trust that this will always work because to my knowledge using ‘is’ should be the way to do it, and that isn’t working.</p>
<p>Can anyone clarify if that is a bug or a feature? How are you supposed to compare two tensors by reference?</p>

tom: <p>Sorry, but what you’re doing is completely bogus. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<pre><code class="lang-python">a = torch.tensor([1, 2, 3])
b = torch.tensor([4, 5])
print(id(a.data)==id(b.data))
</code></pre>
<p>gives <code>True</code>, too!</p>
<p>What happens is that <code>a.data</code> and <code>b.data</code> don’t simultaneously exist, but are newly allocated and immediately disposed of objects, and share the same memory address (see documentation of <code>id</code>).</p>
<p>a and b will be distinct Tensors using to the same memory area, <code>a.data</code> and <code>b.data</code> are completely distinct (also from <code>a</code> and <code>b</code>) tensors, too.<br>
I will add "don’t use <code>.data</code>" – see the PyTorch 0.4 migration guide.</p>
<p>Best regards</p>
<p>Thomas</p>

Florian_Dietz: <p>I see. Thanks for the clarification.</p>
<p>Is there any way to do what I need, though:</p>
<p>Is there a way to compare a and b from the example that will evaluate to True even though b is the detached version of a?</p>

tom: <p>It is not possible to do this exactly.</p>
<p>Well, you apparently don’t want <code>(a == b).all().item()</code>.</p>
<p>What you could do is to see whether they point to the exact same memory range - i.e. <code>.data_ptr()</code>, <code>.storage_offset()</code>, <code>.shape</code> and <code>.strides()</code> are the same.</p>
<p>I’m not sure I recommend that. My impression is that you’re trying to do something that will end up not working well.</p>
<p>Best regards</p>
<p>Thomas</p>

Florian_Dietz: <p>I don’t want to use (a == b).all().item() because it’s possible that two tensors contain the same values by accident, especially since a few tensors get initialized to be all-zeros.</p>
<p>I ended up just storing the original vector along with the detached version in any place where I need to make comparisons. It’s a bit rendundant, but at least this is guaranteed to work.</p>
Index 3D tensor over multiple rows for each main element

acobobby: <p>Hello all,</p>
<p>I  have a 3D tensor called <code>x</code> of size <code>(batch_size, sequence_len, features)</code>.<br>
For each batch element I want to take 3 rows, whose indexes could be different from batch element to batch element.<br>
With a classic for loop over batch dimension it would look like:</p>
<pre><code class="lang-auto">res = torch.empty(x.size(0),3,x.size(2)) # take 3 rows for each batch
for i in range(x.size(0)):
    res[i] = x[i][ rows[i] ] # select appropriate rows
</code></pre>
<p>where <code>rows</code> is a <code>LongTensor</code> containing the indexes of the 3 rows to be taken for each batch element.</p>
<p>Is there a way to get rid of the for loop? I tried many ways but I am not able to make it work for a 3D tensor.</p>
<p>Thank you!</p>

acobobby: <p>For all of you that are interested, I think I found a solution, after many trials and errors:</p>
<p>the trick is to set the <code>torch.LongTensor</code> used as index to have the same 0-th dimension value for each dimension of the original tensor.</p>
<p>The code is very simple:</p>
<pre><code class="lang-auto"># select first and second row for each batch element
rows = torch.LongTensor( [ [0,1]*batch_size ] ) 
# create an index for each batch element
idx = [
        torch.LongTensor(range(batch_size)).unsqueeze(1), 
        torch.LongTensor(rows) 
      ]
# index the tensor
x[idx] # shape (batch_size, 2, features)
</code></pre>
<p>The <code>unsqueeze(1)</code> creates a tensor like this: <code>[[0],[1],[2]...,[batch_size-1]]</code>.<br>
By simply changing <code>[0,1]</code> inside the <code>rows</code> tensor it is possible to get different rows for each batch element.</p>
Is there some tips or techniques for optimizing speed in pytorch?

Ran: <p>My codes runs slowly, except for better gpus, is there any other ways to make codes run faster?<br>
like how to load data faster?</p>

acobobby: <p>A general rule is to avoid explicit for loops when possible.</p>
<p>When manipulating tensors for example, try to vectorize all you can by using the default pytorch functions instead of looping through dimensions.</p>
Issue with torch.stack()

cane95: <p>Hello everyone.</p>
<p>I’m having problems with torch.stack(), I reported a small example with the error that I get</p>
<pre><code class="lang-auto">import torch
a = torch.zeros(3,2,1,2)
b = torch.zeros(2,2,1,2)
torch.stack([a,b], dim=0)
</code></pre>
<pre><code class="lang-auto">RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 0. Got 3 and 2 in dimension 1 at /opt/conda/conda-bld/pytorch_1550780889552/work/aten/src/TH/generic/THTensorMoreMath.cpp:1307
</code></pre>
<p>why do I have this error message?</p>

acobobby: <p>As you can see from the <a href="https://pytorch.org/docs/stable/torch.html?highlight=stack#torch.stack" rel="nofollow noopener">official documentation</a> <code>torch.stack</code> requires tensors of the same size.<br>
The output will be a tensor with a new extra dimension in position 0 with value equal to the number of tensors stacked.</p>
<p>In your case dimension 0 of your original tensors (which is dimension 1 of the resulting tensor) does not match.</p>
<p>Maybe you wanted to concatenate the tensors, in this case you can try with:</p>
<pre><code class="lang-auto">torch.cat([a,b]) # output size is (5,2,1,2)
</code></pre>
ResNet18 - access to the output of each BasicBlock

bl0up: <p>Hello,</p>
<p>I’m using ResNet18 from torchvision and I need to access the output of each <code>BasicBlock</code> in the four <code>layers</code>, i.e the output of <code>bn2</code> of each <code>BasicBlock</code> in the following example.</p>
<pre><code class="lang-auto">(layer1): Sequential(
    (0): BasicBlock(
      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace)
      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    )
    (1): BasicBlock(
      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace)
      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    )
  )

</code></pre>
<p>Thanks a lot.<br>
Ambroise</p>

ptrblck: <p>Have a look at <a href="https://discuss.pytorch.org/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/6">this post</a> to see how to use forward hooks to get the output of some internal modules.</p>

bl0up: <p>Perfect, it works,  thank you!</p>
Is torch.nn.BCELoss `reduction` applied for "pixel" or "batch"?

Jeeyung_Kim: <p>Hi,</p>
<p>I am wondering how the result would be calculated if I set <code>reduction=mean</code> in torch.nn.BCELoss when I use auto-encoder model.<br>
Do I average the loss only for “batch” dimension or both “pixel” and “batch”?</p>
<p>If I average for both “pixel” and “batch”, is there no way to sum loss for all “pixel” and to average for “batch” dimension?</p>
<p>Thanks</p>

MariosOreo: <p>Hello,</p>
<p><code>reduction=mean</code> in BCELoss means average in pixel dimension, the result you get is a scalar.<br>
If you want to sum loss in pixel dimention and perform averaging in batch dim, you could set <code>reduction='none'</code> and manually add sum and average operation.</p>
<pre><code class="lang-auto">criterion = nn.BCEWithLogitsLoss(reduction='none')
...
loss = criterion(logits, target).sum(dim=(1,2,3)).mean(dim=0)
loss.backward()
</code></pre>

Jeeyung_Kim: <p>It is clear. Thanks a lot <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> !!!</p>

Jeeyung_Kim: <p>If I use <code>reduction=mean</code> in BCELoss, you said that the result is a scalar.<br>
and does it mean that it is reduced(mean) in both batch dimension and pixel dimension?</p>
To modify last layer in pytorch

SHASHANK_KUMAR_MISHR: <p>I have to modify the last layer of resnet18 in pytorch where the size of labels=(2,10).<br>
I tried to do that but I am getting errors because of labels size .How to resolve it.<br>
My code for that:-</p>
<p>in_features = resnet.fc.in_features<br>
resnet.fc = nn.Linear(in_features, labels)<br>
Here the size of labels is creating problems as it is (2,10).?</p>

iArunava: <p>Hi <a class="mention" href="/u/shashank_kumar_mishr">@SHASHANK_KUMAR_MISHR</a>!<br>
Welcome to the PyTorch Forums <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Please wrap your code around ``` helps in readability <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Coming to your question,<br>
Why are your labels <code>(2, 10)</code> ?<br>
Labels are the number of classes it should be a scalar value.</p>

SHASHANK_KUMAR_MISHR: <p>I have a problem statement in which image is given in which some text are written.I have to design it in such a manner that it predicts vowels (10 different vowels) and consonants(10 different consonants) simultaneously.That is the reason the shape of calss is (2,10).2 stands for  vowels and consonants each of size 10.I am attaching pic for more clarity<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/0/0102bcff6398e510cd66fc95a06b57130c532140.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/0102bcff6398e510cd66fc95a06b57130c532140" title="Capture"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/0/0102bcff6398e510cd66fc95a06b57130c532140_2_690x386.jpeg" alt="Capture" data-base62-sha1="8Wll4lOSSD9AIWsQi08wmcRrz2" width="690" height="386" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/0/0102bcff6398e510cd66fc95a06b57130c532140_2_690x386.jpeg, https://discuss.pytorch.org/uploads/default/optimized/2X/0/0102bcff6398e510cd66fc95a06b57130c532140_2_1035x579.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/0/0102bcff6398e510cd66fc95a06b57130c532140.jpeg 2x" data-dominant-color="F3F5F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture</span><span class="informations">1118×626 42.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Relation between size of a tensor and nan as result

zahra: <p>Hi,</p>
<p>When I replace this code:</p>
<pre><code class="lang-auto">w1 = torch.Tensor([[np.random.randn(input1.size(2), input1.size(2)) * 5.66]])
</code></pre>
<p>by this code:</p>
<pre><code class="lang-auto">temp = input1.size(2)
w1 = torch.Tensor([[np.random.randn(temp, temp) * 5.66]])
</code></pre>
<p>the output of my program changes to nan !!!<br>
I replace size by shape but there exists such problem.</p>
<p>Do you think what the problem is?</p>
<p>Thanks</p>
`.backward()` for !scalar values is not supported. Is there some workaround other than a loop?

iArunava: <p><code>.backward()</code> is not supported for values other than scalar ones.<br>
Eg:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; bs = 3
&gt;&gt;&gt; n1 = torch.randn(bs, 3, 224, 224)
&gt;&gt;&gt; net = model()
&gt;&gt;&gt; f = net(n1)
&gt;&gt;&gt; f.backward()
. . .
.
RuntimeError: grad can be implicitly created only for scalar outputs
</code></pre>
<p>But</p>
<pre><code class="lang-auto">&gt;&gt;&gt; f[0, 0].backward(retain_graph=1)
</code></pre>
<p>works<br>
so,</p>
<pre><code class="lang-auto">&gt;&gt;&gt; for i in range(bs): f[i, 0].backward(retain_graph=1)
</code></pre>
<p>Is the only solution, right?</p>

Tony-Y: <pre><code class="lang-auto">f.sum().backward(retain_graph=1)
</code></pre>
<p>d(sum_k F(x_k))/dx_j = dF(x_j)/dx_j</p>

iArunava: <p>Yeah true! Right! Thanks!</p>
Why is Pytorch faster after the first training (starting loss also lower)?

highDopamine: <p>I notice that Pytorch has slow training process when I train the model for the first time. But the next time the speed is significantly improved. This makes me think that some caching is done after first training for faster inference or data loading in Pytorch.</p>
<p>One important thing I notice is that the starting loss of later training is much lower than the starting loss in the first training. So I wonder if Pytorch has done something with the weight initialization?</p>
<p>I hope somebody will explain to me this kind of phenomenon so I don’t have to worry about my training code.</p>

alex.veuthey: <p>When you say “the next time”, do you mean the next epoch? That would explain exactly what you describe (faster computational time, lower loss)…</p>
<p>After the first epoch, if your dataset fits in memory, the loading times will be much faster if you are on Linux, because of the system caching system.</p>
<p>However, if you are talking about launching a new experiment with the same script, there is no justification for the <strong>much lower</strong> loss or faster loading times that I can think of…</p>

highDopamine: <p>Yeah, I’m talking about launching a new experiment with the same script. It’s weird for me because I have observed this phenomenon many times. Sometimes the starting loss is not lower, but sometimes it is. The decrease is so significant that I don’t think the random weight initialization is the cause. The only thing that I can think of is that the Pytorch is saving some state after the last training session and loading it into the new one. But it seems impossible.</p>

alex.veuthey: <p>I don’t know much about PyTorch’s internal behaviour, but I’ve never observed this. Even if you specifically load the previous model in the code, it doesn’t explain the big change in initial performance.</p>
<p>One thing that could explain it is that your dataset has very easy samples and very hard samples, and your dataset is shuffled at each training session, therefore the starting batches could include only or mostly hard samples and in other cases only or mostly easy samples… Do you use shuffle in your DataLoader and could you try several experiments (just stop them after a few batches) without shuffle if you do? That could verify this hypothesis.</p>

highDopamine: <p>Looks like you’ve got it right. I think it is because the loss function I use is a square function so if the beginning batches are hard batches the loss will be extremely high and vice versa for the easy batches. Also, my data is not balanced so maybe it causes the oscillation in the starting loss.</p>

eadomenech: <p>In my case, what happens is that pytorch only uses a core, then starts using all the cores that I have.</p>
Multi-task learning: backward pass on intermediate loss?

AIJoris: <p>Hi,</p>
<p>I’m using a seq2seq transformer network in a multi-task learning setting. I have a main text generation task and an auxiliary classification task that uses intermediate output as prediction. For both tasks, I do a full forward pass, but for the auxiliary task I only use the output of an intermediate layer to compute the loss.</p>
<p>My question is what happens when I do a backward pass on this intermediate loss. I would think the forward pass builds a computational graph for the entire network, but since the loss only uses part of this graph I assume the parameters that come after this intermediate layer will not be affected. Is this reasoning correct?</p>

MariosOreo: <p>Hello,</p>
<p>AFAIK, performing backward pass on the full loss is equal to backward separately.<br>
The following two case are the same.</p>
<pre><code class="lang-auto">loss = loss1 + loss2
loss.backward()

loss1.backward(retain_graph=True)
loss2.backward()
</code></pre>
<p>If I am wrong, please correct me.</p>

AIJoris: <p>Thanks for you answer. However, the crux of my question is whether using intermediate network outputs to compute a loss and perform backprop, <em>while doing a full forward pass</em>, will or will not update parameters that influence outputs <em>after</em> my intermediate loss.</p>

Ali_Mirzaeyan: <p>Hi,<br>
I don’t think these two expressions are the same. because the loss is just a number, so when you backpropagate the addition of these two losses, the auxiliary loss also will propagate into the entire network which is not what Joris intended to do. So I think the second one make more sense for the discussed problem.</p>
About dice loss

yw_tt: <p>class BinaryDiceLoss(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(BinaryDiceLoss, self).<strong>init</strong>()</p>
<pre><code>def forward(self, input, target):  #  input.size = target.size and size=(N,1,128,128,128)
    N = target.size(0)   # N is batchsize
    smooth = 1

    input_flat = input.view(N, -1)
    target_flat = target.view(N, -1)

    intersection = input_flat * target_flat

    loss = 2 * (intersection.sum(1) + smooth) / (input_flat.sum(1) + target_flat.sum(1) + smooth)
    loss = 1 - loss.sum() / N

    return loss
</code></pre>
<p>Hello, I train a semantic segmentation model and I use above BinaryDiceLoss to backward model parameters. But  my model loss does not decline. The data input to the model is correct because loss is decline when I user another loss to backward. I want to know how to deal with this problem?  Thank you. I want to know if above BinaryDiceLoss is correct?</p>
How to transfer tf.layers.dense() to pytorch?

Donggeun: <p>How to transfer tf.layers.dense() to pytorch?</p>
<pre><code class="lang-auto">tf.layers.dense(post_outputs, hp.num_freq)
</code></pre>

vabh: <p>Hi,</p>
<p>If you’re looking for the equivalent of <code>tf.layers.dense</code> it is <code>nn.linear</code><br>
<a href="https://pytorch.org/docs/stable/nn.html?highlight=nn%20linear#torch.nn.Linear" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/nn.html?highlight=nn%20linear#torch.nn.Linear</a></p>

Donggeun: <p>Thank you!!<br>
I found it!</p>
How to exclude Embeddings when saving a model

Vladimir_Getselevich: <p>How can I save trained Pyrotch excluding embeddings level (in NLP task) which takes most of the place. I want to recreate this level during inference stage and do not want to save it with the model.</p>

rashmi: <p>I think you could just pop embedding matrix out of model.state_dict() and put it back in for inference.</p>
How to avoid gradient vanishing in CNN

Mrobot: <p>I’m trying to build a basic architecture for super-resolution (can be extended to any image processing program), but I found that the network is hardly trained. The program can be found at <a href="https://github.com/psychopa4/Basic-Architecture-for-Super-Resolution" rel="nofollow noopener">https://github.com/psychopa4/Basic-Architecture-for-Super-Resolution</a></p>
<p>Here is the evaluation during training after several epochs:</p>
<pre><code class="lang-auto">Scale= 8
Dataset= E:/data/DIV2K/sp_val
PSNR_predicted= 23.6633691875079
PSNR_bicubic= 23.679716005992653
It takes average 0.004361677169799805s for processing
</code></pre>
<p>As you can see above, the output of the network is basically the Bicubic upsampling counterparts of the low-resolution images, even worse. I have tried training the network for hundreds of epochs, however, the output tend to converge to the Bicubic upsampling version.</p>
<p>I used to build my model on TensorFlow and it worked well, so I wonder whether I have missed something in the training process with PyTorch. I think it’s due to gradient vanishing, because the network is hardly trained after 3 or 4 epochs. I tried torch.nn.utils.clip_grad_norm but i didn’t work, maybe it’s more suitable for gradient exploding rather than vanishing.</p>
<p>Sorry I didn’t insert code directly, cause I have no idea where the crucial part is.I’d really appreciate it if you could take a few seconds at this program!</p>
<p><strong>About Super-Resolution</strong><br>
Normally, model for super-resolution adopts LR image to reconstruct HR image.</p>
<p>The output of the network is <em>F(I_lr)+I_bic</em>, and the loss function is <em>L1(HR,out)</em>, so the output should be close to HR instead of Bicubic.</p>
<p>This ‘+Bicubic’ strategy is universally adopted for super-resolution.</p>

yw_tt: <aside class="quote no-group" data-username="Mrobot" data-post="1" data-topic="15158">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/94ad74/48.png" class="avatar"> Mrobot:</div>
<blockquote>
<p>Bicubic</p>
</blockquote>
</aside>
<p>how do you deal with this problem? I have met same problem.</p>
Converting List[T] to Tuple[T] in torchscript

Daniel_Pineo: <p>I’m struggling to figure out how to do this, if it’s possible at all.  I have a torchscript function that computes a tensor on each iteration.  Because the tensors have different shapes, it’s not convenient to simply concatenate the tensors, so I’m collecting them in a list.  Because only tuples can be returned from torchscript functions, I’m trying to convert the final list to a tuple to return it.  However, the usual python tuple(my_list) constructor doesn’t seem to work, yielding the error “expected a value of type Tensor for argument ‘0’ but found Tensor<span class="chcklst-box fa fa-square-o fa-fw"></span>”.  Apparently it’s trying to place the entire list into the first element of the tuple, rather than expanding it into a tuple of the same length as the list, as would be expected with the usual python semantics.</p>
<p>Here is some example code to show what I’m trying to do:</p>
<pre><code class="lang-auto">import torch

@torch.jit.script
def f(x):
    
    list_of_tensors = []
    for n in range(3):
         list_of_tensors.append(torch.ones([10]))
    return tuple(list_of_tensors)

traced_script_module = torch.jit.trace(f, [])
</code></pre>
<p>The error I get in this case is</p>
<blockquote>
<hr>
<p>RuntimeError                              Traceback (most recent call last)<br>
 in <br>
1 import torch<br>
2<br>
----&gt; 3 <span class="mention">@torch.jit.script</span><br>
4 def f(x):<br>
5</p>
<p>~/anaconda3/lib/python3.6/site-packages/torch/jit/<strong>init</strong>.py in script(obj, optimize, _frames_up, _rcb)<br>
822     else:<br>
823         ast = get_jit_def(obj)<br>
 → 824         fn = torch._C._jit_script_compile(ast, _rcb, get_default_args(obj))<br>
825         # Forward docstrings<br>
826         fn.<strong>doc</strong> = obj.<strong>doc</strong></p>
<p>RuntimeError:</p>
<p>for operator (Tensor 0) → Tensor:<br>
expected a value of type Tensor for argument ‘0’ but found Tensor<span class="chcklst-box fa fa-square-o fa-fw"></span><br>
Empty lists default to List[Tensor]. Use torch.jit.annotate(List[my_type], <span class="chcklst-box fa fa-square-o fa-fw"></span>) to create an empty list of another type<br>
<span class="mention">@torch.jit.script</span><br>
def f(x):</p>
<pre><code>list_of_tensors = []
for n in range(3):
     list_of_tensors.append(torch.ones([10]))
return tuple(list_of_tensors)
             ~~~~~~~~~~~~~~~ &lt;--- HERE
</code></pre>
<p>:<br>
<span class="mention">@torch.jit.script</span><br>
def f(x):</p>
<pre><code>list_of_tensors = []
for n in range(3):
     list_of_tensors.append(torch.ones([10]))
return tuple(list_of_tensors)
       ~~~~~ &lt;--- HERE
</code></pre>
</blockquote>
Add a fourth input channel to resnet18. Or: Why does copying weights between nn.Conv2d result in a non-leaf node?

bdennyw: <p>I’m trying to add a fourth channel to a pre-trained Resnet.<br>
When I copy the weights over, the Parameter loses its leaf node characteristics.<br>
I’ve tried to set needs gradients etc. but no luck.<br>
What do i need to do to make the newly copied weights a leaf_node ?</p>
<p>Thanks!</p>
<p>Here’s the example code:</p>
<pre><code class="lang-auto">model_ft = models.resnet18(pretrained=True)
#load the weigths into the new convolution, copy the last plane of the orig into the new
new_input_conv = nn.Conv2d(4, 64, kernel_size=7, stride=2, padding=3, bias=False)
new_input_conv.weight[:,0] = model_ft.conv1.weight[:,0].clone()
new_input_conv.weight[:,1] = model_ft.conv1.weight[:,1].clone()
new_input_conv.weight[:,2] = model_ft.conv1.weight[:,2].clone()
new_input_conv.weight[:,3] = model_ft.conv1.weight[:,2].clone()

new_input_conv.weight.is_leaf
</code></pre>
<p>Out[1]: False</p>

bdennyw: <p>Ok I figured this out. For some reason the weights need to be detached and reassigned as a Parameter with grad re-enabled.</p>
<p><code>new_input_conv.weight = nn.Parameter(new_input_conv.weight.detach().requires_grad_(True))</code></p>
nn.dataParallel crashes

jmaronas: <p>Hi.</p>
<p>I am facing a strange problem. When running a code on two GPUs my machine, after some epochs, reboots. If I use the same code running only on 1 GPU everything works fine.</p>
<p>I have follow this old thread <a href="https://discuss.pytorch.org/t/nn-dataparallel-model-cuda-stuck/8085/12" class="inline-onebox">nn.DataParallel(model).cuda() stuck</a> but it does not work.</p>
<p>Any suggestions?.</p>

JuanFMontesinos: <p>Is your power supply powerful enough to feed them?<br>
If it’s on the limit, it’s  a matter of time to have a power peak. In that case, power supply just disconnect current and that causes a “reboot”</p>
<p>Oh, and realise official gpu comsumption is an average rather than a maximum. You may have peaks 1.6 times bigger than specifications</p>
Building the network in pytorch

Rojin: <p>I am trying to implement the following model in torch. I have found source code for Tensorflow but not Torch.  If there is any source that I can read to help me build the model in Torch that would be great.</p>
<p>basically, I need to know:</p>
<ol>
<li>how to feed two inputs</li>
<li>how to specify connections similar to the image</li>
</ol>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/8/832e2e6408c04f5dfbb3503180828d15888ba949.png" data-download-href="https://discuss.pytorch.org/uploads/default/832e2e6408c04f5dfbb3503180828d15888ba949" title="19%20AM"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/8/832e2e6408c04f5dfbb3503180828d15888ba949_2_304x500.png" alt="19%20AM" data-base62-sha1="iItut1VD1aa2t1BzydcDtHvbRqN" width="304" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/8/832e2e6408c04f5dfbb3503180828d15888ba949_2_304x500.png, https://discuss.pytorch.org/uploads/default/optimized/2X/8/832e2e6408c04f5dfbb3503180828d15888ba949_2_456x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/8/832e2e6408c04f5dfbb3503180828d15888ba949_2_608x1000.png 2x" data-dominant-color="ECECEC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">19%20AM</span><span class="informations">663×1087 250 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Thank you!</p>

JuanFMontesinos: <p>Hi, could you explain little bit more what do you want to do? view1 and 2 are your inputs (just by  arrow direction…)?</p>

Rojin: <p>Sure! thanks:-)</p>
<p>View1 and View2 are the two inputs. Both inputs are fully connected to one of the middle layers ( the right one), and to separate neurons in the other latent space ( the left one, you can see that view 1 is connected to the right neuron in the left box, and view 2 is connected to the left neuron in the left box)</p>

JuanFMontesinos: <p>Maybe <a class="mention" href="/u/ptrblck">@ptrblck</a> may help you in a better way but I think what  you can do is building up the whole model as a single nn.Linear layer and  to set gradients equal to 0 for the corresponding neurons in the backward pass (obviously initializing that weight as zero too)</p>
<p>It’s a little bit mess because weights are passed as a package, so you cannot disable requires grad in a subgraph.</p>
<p>Another option could be to initialize corresponding weights to 0 and  detaching specific neurons during the forward pass.</p>
<p>As you can see it’s a mess.</p>
The tutorial of LSTM does not init the hidden state

Robot_mantou: <p>In the pytorch tutorial about <a href="https://pytorch.org/tutorials/beginner/nlp/sequence_models_tutorial.html#sphx-glr-beginner-nlp-sequence-models-tutorial-py" rel="nofollow noopener">LSTM</a>, the code:</p>
<pre><code class="lang-auto"> # Step 1. Remember that Pytorch accumulates gradients.
 # We need to clear them out before each instance
model.zero_grad()

 # Step 2. Get our inputs ready for the network, that is, turn them into
# Tensors of word indices.
sentence_in = prepare_sequence(sentence, word_to_ix)
targets = prepare_sequence(tags, tag_to_ix)

# Step 3. Run our forward pass.
 tag_scores = model(sentence_in)
</code></pre>
<p>does not init the hidden state when the new sentence is coming. Is it right?</p>
Accessing an image tensor with floating point indices

prash6392: <p>Hello everyone !</p>
<p>I have an image tensor of dimension (batch x channels x height x width) and a floating point index (x’, y’).</p>
<p>Is there a way to access the corresponding interpolated pixel value in the image? i.e. I would like to do something like the following.</p>
<pre><code class="lang-auto">bs, nc, h, w = 1, 3, 256, 256
image_tensor = torch.randn(bs, nc, h, w)
x, y = 5.6, 128.4
interpolated_pixel_value = image_tensor[0, :, y, x]
</code></pre>
<p>Essentially, I would like to do a texture look up with floating point co-ordinates in an “auto-diff”'able way. I looked up nn.Embedding, but it seems to me that it supports only integer indices.</p>
<p>Sincerely appreciate your time &amp; help !  <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ptrblck: <p>I think <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.functional.grid_sample" rel="nofollow noopener">grid_sample</a> might work for your use case. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

prash6392: <p>That works perfectly. Thank you so much !</p>
LSTM autoencoder tutorial, please!

codingmonster: <p>I am looking for any codes or tutorials about LSTM autoencoder written with Pytorch but only one can be found from <a href="https://discuss.pytorch.org/t/lstm-autoencoder-architecture/8524/2">here</a> from pytorch discuss section and it’s not that really helpful for me. Any othere source of LSTM autoencoder would be very appreciated!</p>
PyTorch DataLoader Very Slow First Batch

glenn.jocher: <p>We have a popular PyTorch YOLOv3 repository (<a href="http://github.com/ultralytics/yolov3" rel="nofollow noopener">github.com/ultralytics/yolov3</a>) which uses the dataloader for multithreaded performance. The first batch in each epoch always takes several times longer than the rest of the batches, and we’ve noticed that the dataloader is loading up <em>far</em> more events than necessary in this first batch (for no apparent reason). For example, if we print the index requested by the dataloader, set batch_size = 4, num_workers = 2, we see that the dataloader loads up 20 images in this first batch (!!!), and then 4 every subsequent batch (as expected).</p>
<p>So our question is why in the world is the dataloader loading up 5X the number of required images the first batch? This is clearly the cause of the slowdown, but the reason for this behavior remains a mystery to us. This has severely detrimental timing effects on smaller datasets.</p>
<pre><code class="lang-bash">/Users/glennjocher/.conda/envs/yolov3/bin/python /Users/glennjocher/PycharmProjects/yolov3/train.py
Namespace(accumulate=1, backend='nccl', batch_size=4, cfg='cfg/yolov3-spp.cfg', data_cfg='data/coco_32img.data', dist_url='tcp://127.0.0.1:9999', epochs=273, evolve=False, img_size=416, multi_scale=False, nosave=False, notest=False, num_workers=2, rank=0, resume=False, transfer=False, var=0, world_size=1)
Using CPU

Reading images: 100%|██████████| 32/32 [00:00&lt;00:00, 168.00it/s]
Model Summary: 225 layers, 6.29987e+07 parameters, 6.29987e+07 gradients

   Epoch       Batch        xy        wh      conf       cls     total   targets      time
0
4
1
5
2
6
3
7
12
8
9
13
10
14
11
15
16
17
18
19
   0/272         0/7     0.148    0.0947      36.5      1.38      38.1        13      7.93
20
21
22
23
   0/272         1/7     0.177     0.131      36.5      1.75      38.5        20      7.07
24
25
26
27
   0/272         2/7     0.177     0.162      36.5      1.87      38.7        24      6.62
28
29
30
31
...
</code></pre>

alex.veuthey: <p>Do you print the index in your custom dataset class? This might be caused by the dataloader preloading data, which actually speeds up your later batches overall…</p>
<p>Could you print the shape of the <code>imgs</code> tensor in the training loop <code>for i, (imgs, targets, _, _) in enumerate(dataloader):</code>? My guess is that the batch size here should be 4.</p>
Help in Multi Label Classification

mirror: <p>Hello everyone ,<br>
I have to solve a problem where an Image is provided with multiple labels ,<br>
There are a total 13 labels , of which each Image can be assigned to 1 or more labels .<br>
could anyone please tell me solution for it ?<br>
and how could I interpret loss and accuracy for such problem statement !</p>
<p>Really Thanks for your time !! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

MariosOreo: <p>I think you could change the sparse label into one-hot format, then considering it as a ‘binary classification problem’.</p>

alex.veuthey: <p><a class="mention" href="/u/mariosoreo">@MariosOreo</a> one-hot won’t help, as “each Image can be assigned to 1 or more labels”.</p>
<p>This is not only multi-label classification, but also multi-class, which is harder. Benchmarks which aim at solving this type of classification are for example Microsoft COCO, Open Images, PASCAL VOC, and many others.</p>
<p>I think you can use cross entropy as loss, to be verified.</p>
<p>For metrics you should check <a href="https://en.wikipedia.org/wiki/Precision_and_recall" rel="nofollow noopener">precision and recall</a> as well as <a href="https://en.wikipedia.org/wiki/F1_score" rel="nofollow noopener">F1-score</a>, which is the harmonic average of P and R for more stability than just taking the mean.</p>

mirror: <p>Thank you for responding <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

narrowsnap: <p>one-hot can help, for example:<br>
one image raw label is  [1, 3, 5]; then transform to one-hot form as [1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0 , 0, 0],<br>
outputs shape is (n, 13), then use BCEWithLogitsLoss as loss function, I think <a class="mention" href="/u/mariosoreo">@MariosOreo</a>’s solution is right.</p>

alex.veuthey: <p>I agree with most of what you say. But your example is actually “three-hot” <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

narrowsnap: <p>haha, you are right. I don’t know how to call that format, similar like one-hot format.</p>

narrowsnap: <p>One more thing, I think mAP is a good metrics for this problem.</p>

alex.veuthey: <p>That also works, but IMHO it’s a worse indicator when facing imbalanced datasets or imbalanced “hardness” of the classes. For example if you have a very easy class and a very hard class, you’ll get a high AP for the first and low AP for the second, which average to a “meh” AP overall. The worst case is you have many very easy classes and few very hard classes, the average will hide the very low performance on the hard classes, but maybe that hard class is very important to you and you don’t know it unless you show the AP for every class…</p>