akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
BCE loss output is not corresponding to calculation by hand

ivprgrmmr: <p>Hi there,</p>
<p>I am using a DenseNet for a two class classification problem. The last layer of my network is a linear layer, so I am using the torch.nn.BCEWithLogitsLoss() loss function.<br>
I am debugging my network since the training loss is not decreasing. I just calculated the BCE by hand and I calculated the BCE using the pytorch function, however the outputs differ.</p>
<p><em>Example:</em><br>
<em>target = tensor([1.])</em><br>
<em>linear_prediction = tensor([-0.5270], grad_fn=)</em><br>
<em>sigmoid_prediction = torch.sigmoid(linear_prediction)</em><br>
<em>print(sigmoid_prediction) → tensor([0.3712], grad_fn=)</em><br>
<em>loss = torch.nn.BCELoss(sigmoid_prediction,target)</em><br>
<em>print(loss) → tensor([0.9910], grad_fn=)</em></p>
<p><em>However, when I calculate the loss using the following function: loss = -ylog(x), with y=1 and x=0.3712, I get a loss of 0.4304.</em></p>
<p>Does anyone have suggestions why the two outputs could be different?</p>
<p>Thanks in advance!</p>

KFrank: <p>Hi Iv!</p>
<aside class="quote no-group" data-username="ivprgrmmr" data-post="1" data-topic="130847" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ivprgrmmr/48/41662_2.png" class="avatar"> ivprgrmmr:</div>
<blockquote>
<p><em>loss = -ylog(x), with y=1 and x=0.3712, I get a loss of 0.4304.</em></p>
</blockquote>
</aside>
<p>Your notation here (“<em>log(x)</em>”) is ambiguous, but you are using the<br>
so-called <em>common logarithm</em> (log-base-10) in your calculation.<br>
You should be using the "natural logarithm* (log-base-e), instead.<br>
The natural logarithm is the correct choice (and is what pytorch is<br>
using).</p>
<p>Note that pytorch’s <code>torch.log()</code> and python’s <code>math.log()</code> give you the<br>
natural logarithm.  (They both use <code>log10()</code> for the common logarithm.)</p>
<p>Best.</p>
<p>K. Frank</p>

ivprgrmmr: <p>Ofcourse, I mixed those two up. Thanks for the reply!</p>
Performance drop when freeze batchnorm. Why?

jhp: <p>Usually, transfer learning freezes the pre-trained model’s <code>BatchNorm running stats</code> (mean and var), so I did that but it achieves worse performance than when it was unfrozen. To be exact, the convergence is fast, but the generalization fails and the performance is not good.<br>
For my knowledge, I think it’s because the domain of the pre-trained model and my dataset are quite different.  Is it right to unfreeze running stat in batchnorm in my use case or do other tricks exist for finetuning?<br>
I’m still suspicious that this would make the pretrained model meaningless (cannot prevent <code>unforgetting</code>)</p>
[JIT] Concatenating two RecursiveScriptModules

robeast: <p>Dear community,<br>
I am working in the field of deploying PyTorch models. Our customers provide us models which have been <code>jit.scipt(...)</code>-ed and saved as <code>*.pt</code> files. This is the common model exchange interface they have defined for us.</p>
<p>The “unholy” thing that I’m trying to achieve now is to concatenate two such scripted models in form of RecursiveScriptModules.</p>
<pre><code class="lang-python">model1 = torch.jit.load('customer_model1.pt')
model2 = torch.jit.load('customer_model2.pt')

print(type(model1) # Gives me &lt;class 'torch.jit._script.RecursiveScriptModule'&gt;
print(type(model2) # Gives me &lt;class 'torch.jit._script.RecursiveScriptModule'&gt;

# ???
</code></pre>
<p>The questions is whether I can somehow concatenate these two models? Every idea is highly appreciated…</p>
<p>Edit: maybe a side note: we use the models only for inference</p>
<p>Thank you in advance!<br>
Best regards,<br>
RB</p>

ptrblck: <p>Would you like to “concatenate” them such that the output of <code>model1</code> would be passed to <code>model2</code> in a sequential way? If so, then you might be able to create a new custom <code>nn.Module</code>, use both of the scripted models there, and script the new “parent model” again (if needed).<br>
Let me know, if I misunderstood your question.</p>

robeast: <p>Dear <a class="mention" href="/u/ptrblck">@ptrblck</a>,<br>
thank you for your answer, yes that is exactly what I would like to achieve. However, I’m not sure how to convert even a single <code>RecursiveScriptModule</code> into an <code>nn.Module</code> without it’s original class definition available. I’d highly appreciate some hints please <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Thank you &amp; Best regards,<br>
RB</p>

ptrblck: <p>I had something like this in mind:</p>
<pre><code class="lang-python"># save
model1 = nn.Linear(10, 5)
model2 = nn.Linear(5, 2)

model1 = torch.jit.script(model1)
model2 = torch.jit.script(model2)

torch.jit.save(model1, 'model1.pt')
torch.jit.save(model2, 'model2.pt')


# load
class MyModel(nn.Module):
    def __init__(self, model1, model2):
        super().__init__()
        self.model1 = model1
        self.model2 = model2
        
    def forward(self, x):
        x = self.model1(x)
        x = self.model2(x)
        return x

model1 = torch.jit.load('model1.pt')
model2 = torch.jit.load('model2.pt')
model = MyModel(model1, model2)

x = torch.randn(1, 10)
out = model(x)
print(out.shape)
&gt; torch.Size([1, 2])
</code></pre>
<p>I.e. just loading the models and using them wrapped in another parent model.</p>

robeast: <p>Dear <a class="mention" href="/u/ptrblck">@ptrblck</a>,<br>
this is exactly what I wanted to achieve, I didn’t know it was possible this way.</p>
<p>Thank you very much,<br>
RB</p>
Complile pre-built package for windows

pytorch-learning: <p>Hi. I am struggling to compile many pre-build packages for windows, which are compiled with different cuda versions and torch versions.</p>
<p>example:</p>
<ul>
<li>package_name-package_version-torch1.8.0-cu102-cp36-cp36m-win_amd64.whl</li>
<li>package_name-package_version-torch1.8.0-cu110-cp36-cp36m-win_amd64.whl</li>
<li>package_name-package_version-torch1.8.0-cu111-cp36-cp36m-win_amd64.whl</li>
</ul>

ptrblck: <p>I’m not sure I understand the issue correctly, so could you explain what “compile pre-built packages” would mean? Are you having trouble installing the Windows pip wheels or building wheels from source?</p>

pytorch-learning: <p>Thanks for your reply. I develop a pakcage based on pytorch, which is supported in Windows. I want to release my pre-build packages (wheel) with differenct cuda and pytorch. But I don’t want to manually compile it. And I want to compile in docker but torch don’t provide image for windows (<a href="https://hub.docker.com/r/pytorch/pytorch/tags?page=1&amp;ordering=last_updated" class="inline-onebox" rel="noopener nofollow ugc">Docker Hub</a>).</p>

ptrblck: <p>Since you want to build your own custom wheels you could take a look at the <a href="https://github.com/pytorch/builder">pytorch/builder</a> repository where the conda binaries and pip wheels are built.</p>

pytorch-learning: <p>Thanks for your reply, I will have a try.</p>
How to get the sample indices in inference?

gabrielwong1991: <p>Hi, say I have a dataset that is passed into data loader and call the model to do inference (iterate through the data loader).</p>
<p>Say you use <code>SequentialSampler</code> and using a single GPU/TPU with batch size say 32.</p>
<p>You know that 32 predicted labels in each batch will follow the ascending order of the sample in the dataset so you know which predicted label refers to the dataset index. You stack with each iteration until the end and you have your predicted label which follows the order of the original dataset’s index.</p>
<p>Now if you use distributed training to quickly do inference, how do you know which GPU/TPU is taking the sample indices from the dataset and after getting the predicted label I can merge back to the original dataset? This is because some GPU/TPU may do it quicker than others and will mess with the order of the indices? Or even batch 1 will not send to GPU0, batch 2 to GPU1 etc?</p>

mrshenli: <p>cc <a class="mention" href="/u/vitalyfedyunin">@VitalyFedyunin</a> for data loader questions</p>

VitalyFedyunin: <p>As an option, modify your dataset to add an index as part of the return.</p>

gabrielwong1991: <p>Exactly what I did!  Thanks all</p>

Strawn985: <p>population typically select a <em>sample</em> of individuals or families and <em>compute</em> an estimate of the population inequality <em>index</em> from this <em>sample</em><a href="https://www.walgreenslistens.biz/" rel="noopener nofollow ugc">.</a></p>
How to Convert a String to LongTensor, and Back?

Stanley_C: <p>In Karpathy’s minGPT, he lists the dataloader as needing to return individual examples of LongTensors, but how do I convert String to and from LongTensors?</p>

gphilip: <p>If you could give some examples of the kind of conversion that you are after, then I can try finding a way to do that.</p>
Low Validation Loss but IoU metric is small also

Sarmad_GTU: <p>Hi all<br>
i am working on semantic segmentation and the loss is Focal loss<br>
and when I try to get the validation loss and changing the Gamma value of focal loss to get another loss, I found two cases:</p>
<pre><code>     case1: small validation loss with small IoU metric
     caes2: higher validation loss with higher IoU metric 
</code></pre>
<p>Example: for validation loss 0.07 the iou=80% but for validation loss 0.01 the iou=70%<br>
Why did this happen? I am confused</p>
Is dropout in pytorch a pseudo-random operation?

Xiangtaokong: <p>Is dropout in pytorch a pseudo-random operation?<br>
For example, ‘out’ is a tensor with [n,c,h,w]</p>
<pre><code class="lang-auto">out=self.lrelu(self.conv_1(out))        
out1 = F.dropout(out, 0.1, training=True)
out2 = F.dropout(out, 0.1, training=True)
out3 = F.dropout(out, 0.1, training=True)

print(out1)
print(out2)
print(out3)
</code></pre>
<p>When I run the above code many times. As I wish, out1 out2 out3 are not the same every time.<br>
But out1 in the first time is the same as out1 in the second time. out2 and out3 have the same phenomenon.<br>
Is this correct？<br>
Thank you</p>

ptrblck: <p><code>F.dropout</code> will randomly sample the mask in each call using the internal pseudo random number generator. If you seed the code, you would thus get the same random numbers and the result would be equal for different runs of the script.</p>

Xiangtaokong: <p>Thank you very much~  <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=10" title=":grinning:" class="emoji" alt=":grinning:"></p>

HaFred: <p>Hi, I wonder how does the pseudo-random number generator work in pytorch. As in <a href="https://pytorch.org/docs/stable/generated/torch.rand.html" class="inline-onebox" rel="noopener nofollow ugc">torch.rand — PyTorch 1.9.0 documentation</a>, is there any explicit cpp code in which we can determine its mechanism?<br>
Thank you very much.</p>

ptrblck: <p>On the GPU <code>curand</code> should be used and a Philox PRNG. I’m unsure, how random numbers are created on the CPU, but it seems that on Linux systems <code>/dev/urandom</code> might be used.<br>
<a href="https://github.com/pytorch/pytorch/blob/44cc873fba5e5ffc4d4d4eef3bd370b653ce1ce1/c10/core/GeneratorImpl.cpp">CPU reference</a>, <a href="https://github.com/pytorch/pytorch/blob/c371542efc31b1abfe6f388042aa3ab0cef935f2/aten/src/ATen/cuda/CUDAGeneratorImpl.cpp">CUDA reference</a></p>
TorchScript custom ops with dynamic size() param

lcxywfe: <p>I want to write a custom op like <code>Upsample</code>.<br>
<code>y = nn.Upsample(size=x0.size()[2:], mode='bilinear', align_corners=True)(x1)</code><br>
When I trace it, the TorchScript will record the size as a dynamic size of x0, how can I implement it?<br>
I tried to use <code>IntArrayRef</code> in my custom c++ code, but got some static size number in traced TorchScript.</p>
Grammar in docs/unused variables

stevenjin8: <p>Hello!</p>
<p>I noticed that some of the tutorials could use improvements such as correcting grammar/weird code examples.</p>
<p>For example, <a href="https://pytorch.org/tutorials/intermediate/dist_tuto.html" rel="noopener nofollow ugc">this tutorial</a> uses a mix of single and double quotes.</p>
<p>Also, <a href="https://pytorch.org/tutorials/intermediate/dist_tuto.html" rel="noopener nofollow ugc">this example</a> doesn’t use the <code>CONTEXT_SIZE</code> variable to preprocess data.</p>
<p>I’m wondering if there is any interest in fixing things like this. I’d be more than happy to contribute.</p>

ptrblck: <aside class="quote no-group" data-username="stevenjin8" data-post="1" data-topic="131008">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/stevenjin8/48/30165_2.png" class="avatar"> stevenjin8:</div>
<blockquote>
<p>I’d be more than happy to contribute.</p>
</blockquote>
</aside>
<p>That sounds great and I’m sure your fixes will be happily accepted!<br>
Would you mind creating an issue on GitHub explaining what could be improved and then following up with the PR?</p>
Calculate the mean and standard deviation for a multimodal dataset

cvprogrammer: <p>Hello Everyone,</p>
<p>There is a bit of a confusion about calcluating the mean and std of a dataset. Can you please answer my questions?</p>
<ol>
<li>What is the correct way to calculate the mean and std of a dataset?<br>
a) Calculate the mean and std for the whole dataset before splitting the dataset in to test,train and eval?<br>
b) Split the dataset in to test,train and eval and then calculate the mean and std individually for test, train and eval</li>
<li>In case, if we are using datasets of different modalities. Example (RGB, Infrared, depthmaps etc). How to calculate the mean and std?<br>
a) Individual mean and std for each modality?<br>
b) Mix the image modalities and the calulate the mean and std for all the modalites together?</li>
</ol>
<p>Thank you.</p>

ptrblck: <ol>
<li>
<p>You would usually calculate the stats from the training dataset, since the validation and test sets are “unknown” and using their stats could be considered as a data leak.</p>
</li>
<li>
<p>I would claim it depends on the overall use case, i.e. are you able to use different transformations for each modality during testing (is the modality a separate input you’ll get)? If so, you could calculate and apply different stats to each input type. If you are only getting an image and don’t know the modality, I would claim you should try to use the mixed stats. However, this approach might not standardize the data properly (zero mean and unit variance) if the input modalities have different stats.</p>
</li>
</ol>
Basic question on the use of dropout layer in sequential module

Luca_Guarro: <p>When adding dropout layers to a Sequential module, does it matter whether I set the inplace parameter to True or False? Intuitively it seems like it would be safer to set inplace to True but is it fine if it is also False?</p>

tom: <p>If you can use inplace or not depends on whether the layer that gives you the input for dropout needs is output (i.e. the dropout input) for the backward. This is not always obvious. The flip side is just that promise of inplace: It saves memory by reusing the memory for the activation.<br>
In that sense the default inplace=False is the safer option. Indeed, I would venture that in almost all contexts inplace=True is a premature optimization.</p>
<p>Best regards</p>
<p>Thomas</p>
Feedback loop for custom dataset

lbhm: <p>Hello all,</p>
<p>I would like to implement a custom <code>Dataset</code> that has a feedback loop connected to my training code. In specific, after each epoch I would like to return information about the training progress to the dataset so that I can make changes within the dataset that will (hopefully) improve the training.</p>
<p>Now my question is: Is this possible? I know that at the start of training the PyTorch <code>Dataloader</code> creates independent copies of my <code>Dataset</code> object and passes them to each worker. So changing something in the original dataset I created before the start of training would not work in my understanding as it does not alter the dataset copies in the worker processes. Is there some kind of workaround for this limitation or is the “dataset-training loop” connection meant as a strict one-way street in PyTorch?</p>

ptrblck: <aside class="quote no-group" data-username="lbhm" data-post="1" data-topic="130858">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lbhm/48/30762_2.png" class="avatar"> lbhm:</div>
<blockquote>
<p>So changing something in the original dataset I created before the start of training would not work in my understanding as it does not alter the dataset copies in the worker processes.</p>
</blockquote>
</aside>
<p>This is correct (if you are using multiple workers in the <code>DataLoader</code>), but note that you should still be able to manipulate the underlying <code>loader.dataset</code> after each epoch and before starting a new one as long as you are not using <code>persistent_workers=True</code>.</p>

lbhm: <p>Thank you for your response <a class="mention" href="/u/ptrblck">@ptrblck</a>!</p>
<p>I will test your suggestion today. At the beginning of my current project, I immediately set <code>persistent_workers=True</code> as I assumed that it speeds up the execution. Do you have some experience on how large the performance penalty between turning persistent workers on and off is?</p>

ptrblck: <p>I don’t think a general statement would be useful, as te penalty depends on the latency to create the workers and thus e.g. on the workload in <code>Dataset.__init__</code>. I.e. especially if you are pre-loading data, the penalty might be large, otherwise negligible.</p>
How to compile pytorch and cuda runtime into a bundle

111568: <p>I noticed when you install pytorch via pip, cuda &amp; cudnn dependencies are also inclued in the pytorch binaries. I think it’s great, but when I compile pytorch from source,  the binaries I got still need to link to local cuda libraries. So my question is how can I bundle cuda &amp; cudnn into pytorch binaries when compiling from source? <a class="mention" href="/u/rasbt">@rasbt</a> <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

ptrblck: <p>You could statically link all libraries and use the <a href="https://github.com/pytorch/builder"><code>pytorch/builder</code></a> repository to see how the conda binaries and pip wheels are built.</p>
Behavior of BCEWithLogitsLoss pos_weight

whoab: <p>Let’s say we have a nB x 2048 prediction tensor and nB x 2048 ground truth/label tensor.</p>
<p>Let’s say we plug both of these into BCEWithLogitsLoss.</p>
<p>The docs says:<br>
<code>* **pos_weight** ([*Tensor*](https://pytorch.org/docs/stable/tensors.html#torch.Tensor)*,* *optional*) – a weight of positive examples. Must be a vector with length equal to the number of classes.</code></p>
<p>But, if you make a pos_weight of size: nB x 2048 <strong>instead</strong> of the required 2048, the BCEWithLogitsLoss still accepts the pos_weight. Why is that? What happens when you do the former; does it just weight every single scalar?</p>

KFrank: <p>Hi Whoab!</p>
<aside class="quote no-group" data-username="whoab" data-post="1" data-topic="130958" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/w/4bbf92/48.png" class="avatar"> whoab:</div>
<blockquote>
<p>But, if you make a pos_weight of size: nB x 2048 <strong>instead</strong> of the required 2048, the BCEWithLogitsLoss still accepts the pos_weight.</p>
</blockquote>
</aside>
<p>Please see this post for an explanation:</p>
<aside class="quote quote-modified" data-post="2" data-topic="130651">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/pos-weight-and-weight-parameters-in-bcewithlogitsloss/130651/2">"pos_weight" and "weight" parameters in BCEWithLogitsLoss</a> 
  </div>
  <blockquote>
    Hi Ali! 
Let me answer a couple of your specific questions first and then explain 
how I look at it. 

In this case the current (1.9.0) documentation for <a href="https://pytorch.org/docs/1.9.0/generated/torch.nn.BCEWithLogitsLoss.html" rel="noopener nofollow ugc">BCEWithLogitsLoss</a> 
is wrong (or at least misleading).  Quoting: 

weight (<a href="https://pytorch.org/docs/1.9.0/tensors.html#torch.Tensor" rel="noopener nofollow ugc">Tensor</a>, optional) – a manual rescaling weight given to the loss of each batch element. If given, has to be a Tensor of size nbatch. 

On the contrary, weight can have other shapes – for example, it can 
have the same shape as input.  (As near as I can tell, the precise 
r…
  </blockquote>
</aside>

<p>That post discusses primarily <code>BCEWithLogitsLoss</code>’s <code>weight</code> parameter<br>
but the same comments apply almost unchanged to the <code>pos_weight</code><br>
parameter.</p>
<p>Best.</p>
<p>K. Frank</p>
Construct Pytorch dataset from list

Thabang_Lukhetho: <p>Hi everyone,</p>
<p>I recently came across this approach in <strong>TensorFlow</strong> that creates a <strong>TensorFlow</strong> dataset from a list and allows you to batch your data. I want to do a similar thing in <strong>Pytorch</strong>. I have a dataset that does not fit into memory and I wanted to know if there is an equivalent approach in <strong>Pytorch</strong> dataloaders?</p>
<pre><code class="lang-auto">import tensorflow as tf 
dataset = tf.data.Dataset.from_tensor_slices([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])
dataset = dataset.map(lambda x: x*2)
dataset = dataset.batch(64)
dataset = dataset.prefetch(tf.data.experimental.AUTOTUNE)
</code></pre>

Thabang_Lukhetho: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> may you kindly assist</p>

gphilip: <p>Does</p>
<pre><code class="lang-auto">X = torch.Tensor(range(11))
X = torch.square(X)
</code></pre>
<p>do what you want?</p>

Thabang_Lukhetho: <p>No, I have added more information to the question.</p>

gphilip: <p>I don’t know TensorFlow, and so cannot figure out what that code is supposed to do. If you could describe what it is doing in words, I can try to figure out how to do the same in PyTorch.</p>
[jit] Performing Graph Surgery on RecursiveScriptModules

robeast: <p>Dear community,<br>
I am working on deploying PyTorch models. Our customers provide us models which have been <code>jit.scipt(...)</code>-ed and saved as <code>*.pt</code> files. This is the common model exchange interface they have defined for us.</p>
<p>We are now trying to extract subgraphs of the larger full model graph. I could already carve out single layers from a model, e.g.:</p>
<pre><code class="lang-python">scripted_model = torch.jit.load('sc_model.pt')
for i in scripted_model.named_modules():

    # Avoid hierarchical modules - since their modules are represented as single standalone layers as well
    if sum(1 for _ in i[1].named_children()) != 0:
        continue

    # Save layers one-by-one                                                                                                                                                                                                                                          
    i[1].save(f"sc_model_{i[0]}.pt")
</code></pre>
<p><strong>The question</strong>: is there a ways to carve out multiple subsequent layers in a single <code>*.pt</code> file? Is this even possible?</p>
<p>Of course the validity of the subnet model is important, so that we can call it for inference as normally as the full model would be invoked.</p>
<p>I’d really appreciate some help here…<br>
Thank you &amp; Best regards<br>
RB</p>
Loss cannot backward

shayue111: <p>Hi guys, I’m achieving <code>sentence-BERT</code>, when I want to train it and the loss backward cannot work.</p>
<p><strong>The wrong info:</strong></p>
<p><code>RuntimeError: Trying to backward through the graph a second time, but the saved intermediate results have already been freed. Specify retain_graph=True when calling .backward() or autograd.grad() the first time.</code></p>
<p>Following is my train pipeline:</p>
<pre><code class="lang-python">output1, output2 = s_bert(tokenized_input1, tokenized_input2)
labels = torch.tensor([1, 0, 1])

criterion = ContrastiveLoss()
optimizer = Adam(s_bert.parameters(), lr=1e-3)
s_bert.train()

for epoch in range(10):
    loss = criterion(output1, output2, labels)

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

    print(loss)
</code></pre>
<p>I guess the reason for the error is that the loss is computed with <code>output1</code> and <code>output2</code>, which are from the same model. I should make loss update once, but I don’t know how to make it.</p>
<p>Looking forward your helps, the Best!!!</p>

gphilip: <p>You are making the forward pass once, and using the outputs of this forward pass to compute the loss again and again. Is this what you wanted to do?</p>

shayue111: <p>Actually I need the model to receive two source data at once, and calculate the distance between these pairs. And the distance is the component of my loss function. But during the backward, I hope model’s parameter being updated by the loss only once.</p>

gphilip: <p>This does not answer my question.</p>

shayue111: <p>OK, I made a big mistake. The model forward process is out the <code>for-loop</code>, so the <code>loss</code> always be calculated by the same <code>output1</code> and <code>output2</code>.</p>
<p>When I add the <code>output1, output2 = s_bert(tokenized_input1, tokenized_input2)</code> into the loop, it does work.</p>
When should we use a function to adjust learning rate with training proceeding?

nikiguo93: <p>hi I found that in some tutorials regarding training DNNs, they use learning rate adjusting function. But in some others there is nothing like that. Can anybody tell me when and why to use that function?</p>

gphilip: <p>I don’t think there is a general rule which says how to set the learning rate. As far as I understand, this is something that you learn using trial and error to build up your intuition about the dataset and the network you use.</p>
About average loss

nikiguo93: <p>Hi can anybody tell me when should we use average loss for training? I found that in my case when I used loss = mse (output,label) as the loss for learning curve, it would sometimes increase. But if I used average loss instead, the learning curve looks good. I calculate the average loss with the following:<br>
for epoch in range():<br>
for i, batched_images in enumerate(dataloader):<br>
loss = mse(input,label)<br>
running_loss + = loss.item()<br>
num_images += input.size(0)<br>
…<br>
Aveg_loss = running_loss / num_images</p>
<p>Any information would be appreciated:)</p>

ptrblck: <p>The batch loss could be noisy as some batches could yield a higher loss than the previous one (e.g. if the batch contains “hard” samples). However, since the training would generally decrease the overall loss, calculating the average would smooth the curve.</p>

nikiguo93: <p>Thank you for your reply. That makes sense.</p>
Seperate Batchnorm params of multi-group parameters

lkdci: <p>I want to create an api method to remove weight decay from BatchNorm parameters, given multiple groups of parameters with different optimizer args from the user.<br>
I was thinking of iterating over module.modules() to look for BN layers and keep their parameters identities. Then iterate over the param groups and compare those identities addresses, to create 2N param groups, with and without weight decay.<br>
It seems to work well, can you think of cases that this logic might breaks? a simpler solution?</p>
<pre><code class="lang-python">weight_decay = 1e-4
lr = 0.01
param_groups = [{'params': module.backbone.parameters(), 'lr': lr},
               {'params': module.head.parameters(), 'lr': lr * 10}]
optimizer_param_groups = split_weight_decay_params(module, param_groups, weight_decay)

def split_weight_decay_params(module: nn.Module, param_groups, weight_decay):
    # get all batchnorms param ids 
    batchnorm_types = (nn.BatchNorm1d, nn.BatchNorm2d, nn.BatchNorm3d)
    no_decay_ids = []
    for m in module.modules():
        if isinstance(m, batchnorm_types):
            no_decay_ids.append(id(m.weight))
            no_decay_ids.append(id(m.bias))
    
    # split param groups for optimizer
    optimizer_param_groups = []
    for param_group in param_groups:
        no_decay_params = []
        decay_params = []
        for param in param_group["params"]:
            if id(param) in no_decay_ids:
                no_decay_params.append(param)
            else:
                decay_params.append(param)
        # append two param groups from the original param group, with and without weight decay.
        extra_optim_params = {key: param_group[key] for key in param_group if key != "params"}
        optimizer_param_groups.append({"params": no_decay_params, "weight_decay": 0.0, **extra_optim_params})
        optimizer_param_groups.append({"params": decay_params, "weight_decay": weight_decay, **extra_optim_params})
    
    return optimizer_param_groups
</code></pre>
Google Colab, torch.randperm issue

Moritz: <p>Hello everyone,</p>
<p>I have an issue when using torch.randperm in Google Colab and I cant figure out what is going wrong.</p>
<p>Since Im relatively new to pytorch and neural networks, I want to apologize in advance if I misuse / misunderstand some terms <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
<p>Short version: The list of train images is randomly permutated before every train epoch. Interestingly the random permutations between different experiments are exactly the same for the respective epoch.<br>
So if an experiment (experiment 1) is started, the random permutation of epoch 1might be [10485, 10999, 5432, …].<br>
If I repeat the experiment (experiment 2) the random permutation of epoch 1 is again [10485, 10999, 5432, …].</p>
<p>Below you can find the code:</p>
<p>The function create_epoch_tuples gets called before each epoch to create the list of indices of train images to be used in the respective epoch (here that list is called self.qidxs)</p>
<p>def create_epoch_tuples(self, net):</p>
<pre><code>    idxs2qpool = torch.randperm(len(self.qpool))[:self.qsize]
    self.qidxs = [self.qpool[i] for i in idxs2qpool]
    self.pidxs = [self.ppool[i] for i in idxs2qpool]
    ...
</code></pre>
<p>The images are then returned in <strong>getitem</strong> via:</p>
<p>def <strong>getitem</strong>(self, index):</p>
<pre><code>    output = []
    # query image
    output.append(self.loader(self.images[self.qidxs[index]]))
    ...
</code></pre>
<p>I am opening this question since I am confused whether this behaviour of torch.randperm is “normal” since I expected different permutations also between experiments.<br>
Thanks for any help</p>

eqy: <p>Is it possible that some earlier code is setting a seed manually? e.g.,</p>
<pre><code class="lang-auto">import torch
torch.manual_seed(42)
torch.randperm(20)
</code></pre>
<p>Will give you the same results every time.</p>

Moritz: <p>Thanks for your answer!</p>
Compute JS loss between gaussian distributions parameterized by mu and log_var

Zhipeng_Han: <p>My network outputs 4 tensors with shape <code>BxN</code>, where <code>B</code> is batch size, <code>N</code> is latent dim.</p>
<ol>
<li>
<code>source_mu</code> and <code>source_log_var</code> are <code>mu</code> and <code>log_var</code> of N gaussian distributions</li>
<li>
<code>target_mu</code> and <code>target_log_var</code> are <code>mu</code> and <code>log_var</code> of N gaussian distributions also.</li>
</ol>
<p>I want to measure the JS divergence between those distributions.</p>
<p>Here is my code snippet, can someone help to confirm:</p>
<pre><code class="lang-auto">    def compute_js_loss(self, source_mu, source_log_var, target_mu, target_log_var):
        def get_prob(mu, log_var):
            dist = Normal(mu, torch.exp(0.5 * log_var))
            val = dist.sample()
            return dist.log_prob(val).exp()

        def kl_loss(p, q):
            return F.kl_div(p, q, reduction="batchmean", log_target=False)

        source_prob = get_prob(source_mu, source_log_var)
        target_prob = get_prob(target_mu, target_log_var)

        log_mean_prob = (0.5 * (source_prob + target_prob)).log()
        js_loss = 0.5 * (kl_loss(log_mean_prob, source_prob) + kl_loss(log_mean_prob, target_prob))
        return js_loss
</code></pre>
How to update variables until all batches finished

Haozhi: <p>My optimization problem is as follows:</p>
<pre><code class="lang-auto">a = torch.rand(2, 2, requires_grad=True)
b = torch.rand(1, requires_grad=True)
# f is a very time-consuming function which I don't
# want to do it every iteration.
c = f(b)
for i in range(N_epoch):
    for j in range(N_iteration):
        # loop over all batches
        L = loss(data[j], a, c)
        L.backward()
        # udpate a
        a -= lr_a * a.grad
# update b and calculate c
b -= lr_b * b.grad
c = f(b)
</code></pre>
<p>The function to calculate intermediate variable <code>c</code> is very time-consuming. I want to accumulate the gradient of <code>b</code> obtained from each data batch and update it at the end of every epoch. However, this will cost error because computational graph is destroyed when <code>backward</code> is called. Could anyone give me some advice?</p>

Haozhi: <p>@ <a href="https://discuss.pytorch.org/u/ptrblck">ptrblck</a> Could you please give me some advice? Thanks in advance.</p>

ptrblck: <p>I’m not sure how your use case works exactly, but</p>
<aside class="quote no-group" data-username="Haozhi" data-post="1" data-topic="130850">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/haozhi/48/38390_2.png" class="avatar"> Haozhi:</div>
<blockquote>
<p>However, this will cost error because computational graph is destroyed when <code>backward</code> is called.</p>
</blockquote>
</aside>
<p>to keep the computation graph (and the intermediate tensors) alive, you could use <code>backward(retain_graph=True)</code>.</p>

Haozhi: <p>OK, thanks for your advice. The real forward model is complex and the data is big, so I’m afraid this will cuase memory problem. I will try it anyway.</p>
Extract features from F-RCNN/Mask-RCNN

tomb88: <p>I would like to extract the features after ROI Align in object detection.<br>
Specifically for every box detected (top k boxes), I would like to extract the backbone features, either the FC layer before classification or the layer before that in the backbone.<br>
Can anyone suggest a straightforward way of doing this or provide a short script?</p>
<p>Thanks</p>

spanev: <p>Hi <a class="mention" href="/u/tomb88">@tomb88</a> ,</p>
<p>You can refer to this post:</p>
<aside class="quote quote-modified" data-post="2" data-topic="56330">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/spanev/48/14892_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-access-parameters-in-a-certain-layer-of-pretrained-network/56330/2">How to access parameters in a certain layer of pretrained network?</a> 
  </div>
  <blockquote>
    Hi <a class="mention" href="/u/pinata1337">@pinata1337</a>, 
An easy way to find out it is to print your model: 
&gt;&gt;&gt; print(frcnn)
FasterRCNN(
  (transform): GeneralizedRCNNTransform()
  (backbone): Sequential(
    (0): ConvBNReLU(
      (0): Conv2d(3, 32, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
      (1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (2): ReLU6(inplace=True)
    )
[...]

You can always access the submodules of a module by it’s named attribute (or index when th…
  </blockquote>
</aside>

<p>So in MaskRCNN, you would define a forward hook (find more about it <a href="https://discuss.pytorch.org/t/accessing-intermediate-layers-of-a-pretrained-network-forward/12113">here</a>):</p>
<pre><code class="lang-auto">mrcnn.roi_heads.box_roi_pool.register_forward_hook(hook_fn)
</code></pre>
<p>and your <code>hook_fn</code> would catpure the input.</p>

tomb88: <p>Thanks for the prompt response.<br>
Now I have another issue. I was able to extract the features using a forward hook , but a F-RCNN network generates 1000 proposals at this stage.<br>
The filtering step happens later when calling postprocess_detections(). Here we chose up to k boxes and return only those selected. I would like to also return the features only for these boxes but I don’t see any place where the indexes are returned. A list named “keep” internally but is not returned.<br>
Any suggestion as to how to achieve this?</p>

drapado: <p>Hi <a class="mention" href="/u/tomb88">@tomb88</a>,</p>
<p>I’m also interested in extracted these features in mask rcnn per object. Did you manage to do it in the end? If so, can you say how?</p>

citlag: <p><a class="mention" href="/u/tomb88">@tomb88</a><br>
Hello<br>
I have been encountering the same issue, has anyone solve it?<br>
I am trying to get the features from the box_head but I keep getting the 1000 proposals from RPN, I know the later the RPN filters them by a threshold but I cant figure out how to call them.</p>
Windows Torch.distributed Multi-GPU training with Gloo backend not working

sshuair: <p>On windows, It works with single GPU, but not work for Two or more GPU with following setting.</p>
<ul>
<li>init_method= init_method=r"file://D:/torch-dist-train/test.log"</li>
<li>init_method=r"tcp://10.4.220.11:1000"</li>
</ul>
<p>Can anyone help this?</p>
<pre><code class="lang-python">import os
import torch
import torch.distributed as dist
from torch.multiprocessing import Process
import torch.nn as nn
import torch.optim as optim
from torch.nn.parallel import DistributedDataParallel as DDP

def run_ddp(rank, world_size):
    # create local model
    model = nn.Linear(10, 10).to(rank)
    #.to(rank)
    # construct DDP model
    ddp_model = DDP(model, device_ids=[rank])
    # define loss function and optimizer
    loss_fn = nn.MSELoss()
    optimizer = optim.SGD(ddp_model.parameters(), lr=0.001)

    # forward pass
    outputs = ddp_model(torch.randn(20, 10).to(rank))#.to(rank)
    labels = torch.randn(20, 10).to(rank)#.to(rank)
    # backward pass
    loss_fn(outputs, labels).backward()
    # update parameters
    optimizer.step()

def all_reduce(rank, size):
    """ Distributed function to be implemented later. """
    group = dist.new_group([0, 1, 2])
    tensor = torch.ones(3)
    print('{} : Before allreducce: Rank {} has data {}'\
    .format(os.getpid(), rank,tensor))
    dist.all_reduce(tensor, op = dist.ReduceOp.SUM, group= group)
    print('{} : After allreduce: Rank {} has data {}'\
    .format(os.getpid(), rank,tensor))

def broadcast(rank, size):
    group = dist.new_group([0,1])
    if rank == 0:
        tensor = torch.zeros(3)
    else:
        tensor = torch.ones(3)
    print('{} : Before braodcasting: Rank {} has data {}'\
    .format(os.getpid(), rank,tensor))
    dist.broadcast(tensor, src = 0, group= group)
    print('{} : After braodcasting: Rank {} has data {}'\
    .format(os.getpid(), rank,tensor))


def init_process(rank, size, fn, backend='gloo'):
    """ Initialize the distributed environment. """
    print(123)
    # dist.init_process_group(backend, init_method=r"file://D:/torch-dist-train/test.log", rank=rank, world_size=size)
    dist.init_process_group(backend, init_method=r"tcp://10.4.220.11:1000", rank=rank, world_size=size)
    print(f"rank:{rank}")
    fn(rank, size)

def test():
    size = 1
    processes = []
    for rank in range(size):
        p = Process(target=init_process, args=(rank, size, run_ddp))
        p.start()
        processes.append(p)

    for p in processes:
        p.join()

if __name__ == "__main__":
    import os
    # os.environ['GLOO_SOCKET_IFNAME'] = '10.4.220.11'
    print(dist.is_available())
    test()
</code></pre>
How to represent "terminate episode" for Knapsack problem with Pointer Network?

gemsanyou: <p>I am currently implementing a (<a href="https://paperswithcode.com/method/pointer-net" rel="noopener nofollow ugc">Pointer Network</a>) to solve a simple Knapsack Problem. However, I am bit puzzled on the correct (or common or “best”) way to give the agent the option to stop taking item (terminate episode). Currently I have done it in 2 ways, adding raw dummy features or adding encoded dummy features (dummy features are all zeros). I trained both methods for 500K episodes and evaluate their performance on a single predefine testcase in each episode after adding the gradient. I found that concatinating dummy features with the encoded features yielded higher score earlier, but also scored 0 very often. On the other hand, adding the dummy features to the raw features learned to maximize score very slowly. Therefore, my questions are:</p>
<ol>
<li>Is adding the raw dummy features make the learning slower because additional encoding layer learning?</li>
<li>What is the most correct (or common or arguably best) way to give the agent the option to terminate the episode (in this case stop taking item)?</li>
</ol>
<p>Thank you</p>
Using Pre-trained Transformer Model to Expand Short Sentences to Long Sentences

chandra_sutrisno: <p>Hi all,</p>
<p>I need to build a model that can expand multiple short sentences to multiple long sentences. I was thinking to use a pre-trained Transformer model to do this just like when we want to do a paragraph or text summarization except, in this case, I switched the output and input values. I tried this using <code>t5-base</code>, ran it on Google Colab, and using really minimum data like 10 rows of data, the idea was to see whether it works or not regardless of the output. But I always got errors like below:</p>
<blockquote>
<p>RuntimeError: CUDA out of memory. Tried to allocate 502.00 MiB (GPU 0;<br>
11.17 GiB total capacity; 10.29 GiB already allocated; 237.81 MiB free; 10.49 GiB reserved in total by PyTorch)</p>
</blockquote>
<p>I interpret this error as I did something wrong or my idea did not work. Is there anyone who can suggest how to do this?</p>
<p>Please advise</p>

gphilip: <p>I am not familiar with how PyTorch NLP models work, but one thing I would do in the setup that I am familiar with, when such an error is reported, is to reduce the batch size.</p>

chandra_sutrisno: <p>Hi <a class="mention" href="/u/gphilip">@gphilip</a> ,</p>
<p>That works, thanks.</p>
Validation and training loss almost constant

Istvan_Ferenc_Toth: <p>Hello! I am trying to build a CNN in order to extract from some images a physical quantity, which should be an array, so this is a kind of regression problem.</p>
<p>I have downloaded and used a pytorch version of the DenseNet (<a href="https://github.com/bamos/densenet.pytorch" class="inline-onebox" rel="noopener nofollow ugc">GitHub - bamos/densenet.pytorch: A PyTorch implementation of DenseNet.</a>) modified so that I have before the dense blocks 2 conv layers (kernel sizes 5 and 3) instead of 1 and the last dense block is flattened and is followed by two fully connected layers. The 3 dense blocks contain six conv layers each with growth rates 8, 16 and 32, respectively.</p>
<p>I have tried different learning rates and optimization (adam, mseprop, sgd) and loss functions (mse, l1), but the training loss and the averaged validation loss does not seem to change (in some cases a decrease may be observed for the first few epochs maybe).</p>
<p>I have a dataset of about 10000 items and I split is to training, validation and test data (70%, 15%, 15%) using scikit-learn’s train_test_split(). The training data is shuffled when loaded by the Dataloader. A single loaded item contains a 256x256 image and the target array, which has also 256 element. The image is grayscale image and is transformed to values between 0 and 1 only by the ToTensor() transformation before entering the model.</p>
<p>My suspicion is that the target array should be scaled or normalized too, but I am not sure how, since some of these arrays contains only positive values , others mostly negative and some are in between. Also the values of the elements of the different arrays may differ significantly and even inside a single array the maximum value may be around 0 and the minimum lets say -200. I may also mention that the same model predicts mostly accurately a different physical quantity from the same images but of course different target arrays (element values in this case are between 0 and 1). Can someone suggest a strategy to tackle the first problem ? Many thanks and apologies for the long description.</p>

ptrblck: <p>Normalizing the output could work e.g. if your target has specific bounds.<br>
E.g. in the past I’ve worked on a keypoint detection use case where the target coordinates were in the range <code>[0, 96]</code>. While the model might have been able to learn also this distribution, the training was faster by normalizing the targets to <code>[0, 1]</code> and “de-normalizing” it to <code>[0, 96]</code> to calculate the RMSE as well as the predictions. I don’t know, if you could also apply a similar workflow or of the target range is much larger and the normalization could “squeeze” small values into a tiny range.</p>
Effect of driving display on PyTorch performance

Fred2: <p>I have a computer with both a NVIDIA Geforce RTX 3070 and an Intel UHD Graphics 630.  The Intel chip came disabled, with the manufacturer suggesting that the NVIDIA drive the monitors.<br>
If I use the NVIDIA GPU to drive my displays (Windows desktop, IDE, web browser, etc), is that taking any appreciable amount of resources away from the NVIDIA being able to do PyTorch/CUDA work?  Would it be beneficial to have the Intel GPU run the desktop so that the NVIDIA can be dedicated to PyTorch?<br>
One downside of using the Intel GPU is that it consumes system memory, and uses electricity power.  The power means heat, so the CPU can’t run as hard.</p>
<p>Thanks.</p>

ptrblck: <p>A disadvantage of using your 3070 for the display is that you would be using GPU memory, which is usually more limited than the host RAM.<br>
E.g. in my plain Ubuntu20.04 setup my GPU is using ~1.2GB for the display output (<code>Xorg</code>, <code>gnome-shell</code>, <code>firefox</code> etc. are reported by <code>nvidia-smi</code>).</p>

Fred2: <p>Thanks, that helps.  I didn’t know about nvidia-smi.  I also see that the NVIDIA’s memory is being used.</p>
Faster, Efficient way to indexing and renewing

DDavid: <p>reproducible code :</p>
<pre><code class="lang-auto">import torch
x = torch.rand(3)
print(x)
# tensor([0.5647, 0.7095, 0.4463])

x[x&gt;0.5] = 1
print(x)
# tensor([1.0000, 1.0000, 0.4463])

</code></pre>
<p>I’m looking for a faster/efficient way to do the <code>x[x&gt;0.5] = 1</code>.</p>
<p>Thanks in advance.</p>