akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
The loss will be Nan when I use loss function defined by torch.nn.function.mse_loss

Fangkang515: <p>The loss is always Nan when I use the loss function as follow:</p>
<pre><code class="lang-auto">def Myloss1(source, target):
    loss = torch.nn.functional.mse_loss(source, target, reduction="none")
    return torch.sum(loss).sqrt()

...

loss = Myloss1(s, t)
loss.backward()

</code></pre>
<br>
<p>But when I use the following loss function, the training becomes normal:</p>
<pre><code class="lang-auto">def Myloss2(source, target):
    diff = target - source
    loss = torch.norm(diff)
    return loss
...

loss = Myloss2(s, t)
loss.backward()

</code></pre>
<br>
<p>Why can’t use the ‘Myloss1’ to train？ Aren’t Myloss1 and Myloss2 equivalent?</p>
<p>Please help me，thank you very much！</p>
Training LSTM Network as many-to-one or many-to-many

Johannes_L: <p>Hello!<br>
I have one basic decision to make and don’t really know what’s the better way.<br>
I am using a LSTM Network to classify time series. Later the network should work online so that every input state is classified.</p>
<p>(1): Now, for training I could give a sliding window with the last n timesteps as input (n high enough/suitable for the task of course) and use the last output for the classification of the last time step (many-to-one)</p>
<p>(2): I could also give a big sequence and use every output to classify every timestep in this sequence (many-to-many)</p>
<p>Which is the better way to do it? In (1) I got for every timestep the same length of previous information. In (2), can I use a sliding window? Then every datapoint is used n times in one epoch with n=sliding_window_size.</p>
<p>I’m thankful for any recommendations!</p>
What does adaptive average pooling do and when to use it?

Superklez: <p>I’ve seen the code for the CNN models in torchvision and I’ve noticed that <code>nn.AdaptiveAvgPool2d</code> is used after the convolutional layers before passing the flattened inputs into the fully-connected layers. So why is it used and is the output size of <code>nn.AdaptiveAvgPool2d</code> always set to whatever the output size of the input is after passing it through the convolutional layers?</p>

Eta_C: <p>Actually, <code>nn.Linear</code> need a certain <code>in_features</code>, which is <code>CxHxW</code>. Now you can see <code>H</code> and <code>W</code> depend on the input resolution.<br>
Following the <a href="https://pytorch.org/docs/stable/generated/torch.nn.AdaptiveAvgPool2d.html" rel="noopener nofollow ugc">document</a>, AdaptivaAvgPool2d</p>
<blockquote>
<p>Applies a 2D adaptive average pooling over an input signal composed of several input planes.</p>
<p>The output is of size H x W, for any input size. The number of output features is equal to the number of input planes.</p>
</blockquote>
<p>It is used to fix <code>in_features</code> for any input resolution.</p>
How can I reshape Datafrom BigBatchSize to Just right BatchSize

sanyu_kimura: <p>I want to Inspection picture in Libtorch.<br>
but my data is too big batches to Inspect on GPU like  [9857 , 3, 64, 64]<br>
so I want to split data like  [5000 , 3, 64, 64] and  [4857 , 3, 64, 64]</p>
<p>I tryed “reshape” command but batch count is different</p>
<p>How can I split data by different batch count</p>
<p>thanks</p>

sanyu_kimura: <p>sorry I found “split” command.</p>
<p>so this topic is solved</p>
Visualize A Neural Network Like In Research Papers

Adithya_Sherwood: <p>Hi Everyone so how do I visualize my model which is a Unet Model by the way like the ones in the research papers</p>
<p>if you are confused just tell me how can I do something like this to my network:</p>
<p><img src="https://cdn-images-1.medium.com/max/540/1*Aa2fKFN2PEKmMQoy8ps-lw.png" alt="" width="540" height="279"></p>
<p>Thanks and please reply as soon as possible</p>

Adithya_Sherwood: <p>Could some one please tell me how to do it please?</p>

Eta_C: <p>Check <a href="https://github.com/ashishpatel26/Tools-to-Design-or-Visualize-Architecture-of-Neural-Network" rel="noopener nofollow ugc">Tools-to-Design-or-Visualize-Architecture-of-Neural-Network</a>.</p>
Comparing two tensors without using for loop

melike: <p>Hi all, I need to compare each row of a tensor with corresponding index of another tensor without for loop. Here is an example,</p>
<pre><code class="lang-auto">inp = tensor([[[[[ 6.,  2.,  5.,  6.],
           		 [ 3.,  4.,  7.,  8.]],

      			[[ 9., 10., 14., 14.],
           		 [11., 16., 16., 16.]]]]]) 

maxes = torch.tensor([[[[ 6.,  8.],
     				    [14., 16.]]]])
</code></pre>
<p>Comparison should return,</p>
<pre><code class="lang-auto">tensor([ True, False, False,  True],
       [False, False, False,  True],
       [False, False,  True,  True],
       [False,  True,  True,  True])
</code></pre>
<p>Any ideas on how to achieve that ?</p>

Prashant: <p>I think you want <a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor.expand" class="inline-onebox" rel="noopener nofollow ugc">torch.Tensor — PyTorch 1.7.0 documentation</a>. You can expand maxes to the larger size of the full input and do a comparison</p>

Eta_C: <p>try</p>
<pre><code class="lang-python">result = (inp == maxes[..., None]).squeeze()
</code></pre>
How can find available gpu?

J_Na: <p>Hi all,<br>
Is there any cuda functions that automatically choose one of free gpus?<br>
I have multi-gpus and I allocate one gpu per one program. so everytime I find available gpus using <code>nvidia-smi</code>, and then I choose one using like <code>os.environ[CUDA_VISIBLE_DEVICES]='0'</code>  This is quite uncomfortable, so I want to cuda functions that automatically find free gpu_num and use that.</p>

kelam_goutam: <pre><code class="lang-auto">def get_free_gpu():
    os.system('nvidia-smi -q -d Memory |grep -A4 GPU|grep Free &gt;tmp')
    memory_available = [int(x.split()[2]) for x in open('tmp', 'r').readlines()]
    return np.argmax(memory_available)
</code></pre>
<p>guess this will help.</p>

J_Na: <p><a class="mention" href="/u/kelam_goutam">@kelam_goutam</a> Thanks for reply!<br>
If I already running on gpu:0 and Memory-usage is 100MiB / 10000MiB, this code not allocate on gpu:0 ?<br>
What is the criterion of ‘memory_available’.</p>

lorenz-h: <p><a class="mention" href="/u/j_na">@J_Na</a> I had the same problem and created a package <a href="https://github.com/lorenz-h/nvsmpy" rel="nofollow noopener">available on github</a> that can parse the output of nvidia-smi automatically and returns a list of all unused gpus. The gpus are filtered in python (not by nvidia-smi) so you could redefine “available” to mean whatever criterion you prefer.</p>

yusanshi: <p>You can use this to figure out the GPU id with the most free memory:</p>
<pre><code class="lang-auto">nvidia-smi --query-gpu=memory.free --format=csv,nounits,noheader | nl -v 0 | sort -nrk 2 | cut -f 1 | head -n 1 | xargs
</code></pre>
<p>So instead of:</p>
<pre><code class="lang-auto">python3 train.py
</code></pre>
<p>You can use:</p>
<pre><code class="lang-auto">CUDA_VISIBLE_DEVICES=$(nvidia-smi --query-gpu=memory.free --format=csv,nounits,noheader | nl -v 0 | sort -nrk 2 | cut -f 1 | head -n 1 | xargs) python3 train.py
</code></pre>
<p>You can also add it into your <code>.bashrc</code> then you can directly run <code>python3 train.py</code>:</p>
<pre><code class="lang-auto">alias python3='CUDA_VISIBLE_DEVICES=$(nvidia-smi --query-gpu=memory.free --format=csv,nounits,noheader | nl -v 0 | sort -nrk 2 | cut -f 1 | head -n 1 | xargs) python3'
</code></pre>
How to fix the result during inference at models using batchnorm

Hypernova: <aside class="quote quote-modified" data-post="1" data-topic="75877">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/59ef9b/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/model-shows-different-predictions-after-training-without-weight-update/75877">Model shows different predictions after training without weight update</a> 
  </div>
  <blockquote>
    Dear Community, 
i encountered some none-intuitive behaviour. I load a model, set it to evaluation mode and predict a single image using model(input). Then, I set the model to training mode and predict a single image again. (Note that there is no .backward performed, I even disabled requires_grad for all parameters). When I set the model to evaluation mode again, and predict the same image from before, I get a different result. 
TL;DR: when you eval an input before and after a training where no …
  </blockquote>
</aside>
<p>
I faced the same problem as the above link on CIFAR10/ResNet. My model uses Batchnorm, and its update changes the result.<br>
But I want to make my result fixed. How can I do that?</p>
<pre><code class="lang-auto">
net = ResNet(ResidualBlock, 3, 16, 3, 10).to("cuda")
                net.load_state_dict(torch.load("pretrained/ckpt.pth"), strict=False)
                net.eval()
                
                with torch.no_grad():
                    input = input.to("cuda")
                    output = net(input)
                    _, predicted = output.max(1)
                    print(output)
                    print("%5s" % classes[predicted[0]])
</code></pre>

ptrblck: <p>The <a href="https://discuss.pytorch.org/t/model-shows-different-predictions-after-training-without-weight-update/75877/2">answer</a> from the linked post explains, that the running statistics in batchnorm layers will be updated during training and used during evaluation (<code>model.eval()</code>).<br>
If you want to keep these stats constant, use <code>model.eval()</code> and don’t perform any forward passes while the model is in training mode.</p>

Hypernova: <p>My net was in evaluation mode but I noticed that I didn’t load the optimizer. After loading the optimizer the problem was solved. Thanks!</p>
Tensorflow to Pytorch User

Elad_Warshawsky: <p>I recently switched to Pytorch because of the flexibility and all the research/work that has been done with it. I am finding it difficult to get a hang of, at least for custom loss functions, and all of the datatypes/arrays and how to convert/manipulate them. What are some goods tips/tricks or quick videos/courses that can can help me solve these issues?</p>

KFrank: <p>Hi Elad!</p>
<aside class="quote no-group" data-username="Elad_Warshawsky" data-post="1" data-topic="109383" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/elad_warshawsky/48/30055_2.png" class="avatar"> Elad_Warshawsky:</div>
<blockquote>
<p>What are some goods tips/tricks or quick videos/courses that can can help me solve these issues?</p>
</blockquote>
</aside>
<p>In addition, of course, to studying the pytorch api documentation, you<br>
should take a look at the pytorch tutorials, many of which are good.</p>
<p>You could do worse than start with pytorch’s <a href="https://pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html" rel="noopener nofollow ugc">60 Minute Blitz</a> tutorial.</p>
<p>The kinds of documentation and tutorials that work well for someone<br>
is largely a matter of personal taste, but its worth checking out those<br>
provided by the pytorch project.</p>
<p>Pytorch and tensorflow have similar higher-level concepts and goals,<br>
but differ in their detailed implementations.  So I would recommend<br>
trying to forget the <em>details</em> of tensorflow, as pytorch will likely be different<br>
(but still make use of the concepts you learned working with tensorflow).</p>
<p>And if you have questions after you’ve tried something specific, post<br>
some (complete, runnable) sample code here, and ask questions.</p>
<p>Best.</p>
<p>K. Frank</p>
Transforms - resize equivalent of fiveCrop

Sebastian_E: <p>When training (or specially testing) CNNs, it of course makes a difference if I resize an image to 300px first and then take a 244x244 crop, or if I take the crop out of a 900px image.<br>
I’ve seen the transformation function <code>FiveCrop</code> that takes one image and returns five crops, one from each corner and one from the centre. Is there an equivalent for resize, where I can get five (or N) different resizes out of one image/tensor?<br>
I know that there is <code>RandomResizedCrop</code> that I could use and input an image multiple times, but I would like to have something without randomness.</p>

ptrblck: <p>You could apply the <code>torchvision.transforms.functional.resize</code> method multiple times on the image and return the samples e.g. as a <code>list</code>. Since each sample would have a different size, you would need to write a custom <code>collate_fn</code>, as the default one won’t be able to <code>stack</code> the tensors in the batch dimension.</p>
RTX3080 is even slower than my old 2070 super

wzm: <p>I was running a simple network and found my new RTX 3080 is even slower than my old 2070 super. I checked the time used for each step. The backward propagation is really slow.</p>
<p>I am using pytorch 1.7.1 with CUDA 11.2.</p>

ptrblck: <p>The slowdown might have been introduced by non-trained cudnn heuristics for the 3080, e.g. in cudnn8.0.5. Based on your setup description I assume you’ve built PyTorch from source, since you are using CUDA11.2 while the <code>1.7.1</code> binaries ship with CUDA11.0? If that’s the case, which cudnn version have you used to build it?<br>
Could you also post your model, so that we could check for regressions in internal cudnn builds?</p>

wzm: <p>I use a simple network.</p>
<pre><code class="lang-auto">class Net(torch.nn.Module):
    def __init__(self, D_in=24, H=[128,128,128,128], D_out=1, drate = 0.0):
        
        super(Net, self).__init__()
        self.linear1 = torch.nn.Linear(D_in, H[0], bias= True)
        self.linear2 = torch.nn.Linear(H[0], H[1], bias= True)
        self.linear3 = torch.nn.Linear(H[1], H[2], bias= True)
        self.linear4 = torch.nn.Linear(H[2], H[3], bias= True)
        self.linear5 = torch.nn.Linear(H[3], D_out, bias = True)
        self.dlayer = torch.nn.Dropout(p=drate)
    
    def forward(self, x):
        x = self.dlayer(x)
        x = F.relu(self.linear1(x))
        x = self.dlayer(x)
        x = F.relu(self.linear2(x))
        x = self.dlayer(x)
        x = F.relu(self.linear3(x))
        x = self.dlayer(x)
        x = F.relu(self.linear4(x))
        x = self.dlayer(x)
        x = self.linear5(x)

        return x
</code></pre>
<p>The cudnn verison is 8.04 (torch.backends.cudnn.version() gives 8004).<br>
I used Adam to train the network.</p>
Help to combine: ReduceLROnPlateau + OneCycleLR (CosineAnnealingLR)

VladimirAlekseevich: <p>Good afternoon. Help to combine the two chedulers (I can’t do it) ReduceLROnPlateau + OneCycleLR (CosineAnnealingLR)</p>
<p>optimizer = torch.optim.Adam(model.parameters(), lr=LR)<br>
scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(<br>
optimizer,<br>
factor=0.7,<br>
patience=scheduler_N,<br>
threshold=1e-10,<br>
verbose=True)<br>
LR_shed = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
max_lr = LR/0.7<br>
for epoch in range(model_epoch_start, model_epoch_end):<br>
scheduler_b = torch.optim.lr_scheduler.OneCycleLR(<br>
optimizer,<br>
max_lr=max_lr,<br>
total_steps=len(train),<br>
pct_start=0.3,<br>
cycle_momentum=False)<br>
for x, y in train:<br>
…<br>
model.train()<br>
…<br>
model.eval()<br>
…<br>
LR_shed.append(optimizer.param_groups[0][‘lr’])<br>
scheduler_b.step()<br>
…<br>
scheduler.step(loss_val)<br>
max_lr = scheduler._last_lr[0]/0.7<br>
…<br>
plt.plot(LR_shed)</p>
<p>work example: ReduceLROnPlateau + OneCycleLR</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/f/bf4f11527dda547a92adb934126c8f015ed1642e.png" data-download-href="https://discuss.pytorch.org/uploads/default/bf4f11527dda547a92adb934126c8f015ed1642e" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/f/bf4f11527dda547a92adb934126c8f015ed1642e.png" alt="image" data-base62-sha1="rioB6Or91OvKbtvh0LMouV09lMy" width="690" height="85" data-dominant-color="F5F6F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">899×112 2.12 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Help me</p>
Tf.where equivalent in Pytorch

bdqnghi: <p>An example of Tf.where usage is like this:</p>
<pre><code class="lang-auto">import tensorflow as tf
import os

os.environ["TF_CPP_MIN_LOG_LEVEL"] = "2"

# Constants (3-element arrays).
a = tf.constant([100, 200, 300])
b = tf.constant([1, 2, 3])

# Use placeholder for predicate to where.
# ... We pass in an array of 3 bools to fill the placeholder.
j = tf.placeholder(tf.bool, [3])

# Use where to apply 1 of 2 methods based on each predicate.
# ... First argument is the predicate (contains bools).
#     Second argument is run when true.
#     Third argument is run when false.
x = tf.where(j, a + 5000, a + b)

# Run with 3 bools in placeholder.
array_temp = [False, True, False]
result = tf.Session().run(x, {j: array_temp})

# For false, add 2 elements toe get her.
# ... For true, add 5000 to first element.
print(result)
</code></pre>
<p>Not sure if there is something that can do the same job like tf.where in this case.</p>

ptrblck: <p>I think <a href="https://pytorch.org/docs/stable/torch.html#torch.where">torch.where</a> should do the same as <code>tf.where</code>.</p>

bdqnghi: <p>Yes, just found this as the solution. Thanks !!!</p>

Mohamed_Nabih: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> <a class="mention" href="/u/bdqnghi">@bdqnghi</a></p>
<aside class="quote no-group" data-username="bdqnghi" data-post="1" data-topic="28344">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/d9b06d/48.png" class="avatar"> bdqnghi:</div>
<blockquote>
<p><code>tf.constant([100, 200, 300])</code></p>
</blockquote>
</aside>
<p>Can I ask what is the equivalent pytorch of tf.constant</p>

ptrblck: <p>You could use <code>torch.tensor</code>s directly.</p>
Distributed Training RuntimeError: arguments are located on different GPUs

xdwang0726: <p>Hi, I got this error when using torch.distributed on training a model on a single machine multiple GPUs.</p>
<pre><code class="lang-auto">  File "run_classifier_bertgcn.py", line 364, in &lt;module&gt;
    main()
  File "run_classifier_bertgcn.py", line 319, in main
    train(train_dataset, model, mlb, G, args.batch_sz, args.num_epochs, criterion, device, optimizer, lr_scheduler)
  File "run_classifier_bertgcn.py", line 136, in train
    output = model(input_ids, attention_mask)
  File "/home/xdwang/ml4h/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/xdwang/ml4h/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 511, in forward
    output = self.module(*inputs[0], **kwargs[0])
  File "/home/xdwang/ml4h/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/lustre03/project/6001103/xdwang/MeSH_Indexing_RGCN/model.py", line 349, in forward
    output, _ = self.bert(src_input_ids, src_attention_mask)
  File "/home/xdwang/ml4h/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/xdwang/ml4h/lib/python3.7/site-packages/transformers/modeling_bert.py", line 838, in forward
    input_ids=input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds
  File "/home/xdwang/ml4h/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/xdwang/ml4h/lib/python3.7/site-packages/transformers/modeling_bert.py", line 197, in forward
    inputs_embeds = self.word_embeddings(input_ids)
  File "/home/xdwang/ml4h/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/xdwang/ml4h/lib/python3.7/site-packages/torch/nn/modules/sparse.py", line 126, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/home/xdwang/ml4h/lib/python3.7/site-packages/torch/nn/functional.py", line 1814, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: arguments are located on different GPUs at /pytorch/aten/src/THC/generic/THCTensorIndex.cu:403
</code></pre>
<p>My model is a simple BERT model (using huggingface transformer) with a sigmoid activation after BERT output.</p>

Yanli_Zhao: <p>what is your code to train single machine multiple GPUs?</p>

xdwang0726: <p>yes, it’s on single machine with 4 GPUs</p>

ptrblck: <p>Could you check, if you are creating tensors inside the <code>forward</code> method using <code>.cuda()</code> or <code>to('cuda')</code>?<br>
If so, make sure these new tensors are created on the currently used device by using the <code>.device</code> attribute of any parameter or the input via <code>.to(x.device)</code>.</p>
Pytorch Error occur, when converting my custom model to torch

kcw4875: <p>I trained the model for the experiment, I did a test on pytorch well.<br>
but ,The following error occurs when converting and saving through torch.jit.trace.</p>
<p>Full error phrase :</p>
<p>Traceback (most recent call last):<br>
File “D:/Landmark/pytorch/convert2C_merge_model.py”, line 190, in <br>
traced_script_module.save("./model/output.pt")<br>
File “C:\Users\kcw\Anaconda3\envs\pytorch\lib\site-packages\torch\jit_<em>init</em>_.py”, line 1205, in save<br>
return self._c.save(*args, **kwargs)<br>
RuntimeError:<br>
could not export python function call Scatter. Remove calls to Python functions before export. Did you forget add <span class="mention">@script</span> or <span class="mention">@script_method</span> annotation? If this is a nn.ModuleList, add it to <strong>constants</strong>.:<br>
C:\Users\kcw\Anaconda3\envs\pytorch\lib\site-packages\torch\nn\parallel\scatter_gather.py(13): scatter_map<br>
C:\Users\kcw\Anaconda3\envs\pytorch\lib\site-packages\torch\nn\parallel\scatter_gather.py(15): scatter_map<br>
C:\Users\kcw\Anaconda3\envs\pytorch\lib\site-packages\torch\nn\parallel\scatter_gather.py(28): scatter<br>
C:\Users\kcw\Anaconda3\envs\pytorch\lib\site-packages\torch\nn\parallel\scatter_gather.py(35): scatter_kwargs<br>
C:\Users\kcw\Anaconda3\envs\pytorch\lib\site-packages\torch\nn\parallel\data_parallel.py(159): scatter<br>
C:\Users\kcw\Anaconda3\envs\pytorch\lib\site-packages\torch\nn\parallel\data_parallel.py(148): forward<br>
C:\Users\kcw\Anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\module.py(481): _slow_forward<br>
C:\Users\kcw\Anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\module.py(491): <strong>call</strong><br>
D:/Landmark/pytorch/convert2C_merge_model.py(74): forward<br>
C:\Users\kcw\Anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\module.py(481): <em>slow_forward<br>
C:\Users\kcw\Anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\module.py(491): <strong>call</strong><br>
C:\Users\kcw\Anaconda3\envs\pytorch\lib\site-packages\torch\jit_<em>init</em></em>.py(688): trace<br>
D:/Landmark/pytorch/convert2C_merge_model.py(187): </p>
<p>help me… plz…</p>

eellison: <p>Could you pleas post a repro and/or file a github issue?</p>

tom: <p><a class="mention" href="/u/xsacha">@xsacha</a> seemed to have run into a similar message (also about Scatter) and debugged it to be accidentally converting a DDP model.</p>
How (specifically) does torchtext.data.Field build_vocab function handle assigning integers to tokens when corpus has more words than vocab max_size?

vintagedeek: <p>For example, suppose I have a TEXT Field object and do <code>TEXT.build_vocab(train_set, max_size=50000)</code>. My understanding is that <code>build_vocab()</code> finds all unique words in the training set (which has been tokenized by the tokenizer passed to the Field object) and assigns them a unique integer. So in this case, my vocabulary will consist of 49,998 words mapped to unique integers. The other two words are <code>&lt;unk&gt;</code> (index 0) and <code>&lt;pad&gt;</code> (index 1).</p>
<p><strong>Question</strong><br>
What if my train set has 100,000 unique words? Does <code>build_vocab()</code> only assign unique integers to the first 49,998 unique words encountered (and <code>&lt;unk&gt;</code> and <code>&lt;pad&gt;</code>), or does it also consider how frequently a word occurs? I see a counter parameter in the vocab object docs, but not sure how it is used (also see <code>min_freq</code> which makes sense to me).</p>

Abhilash_Srivastava: <p>Great question!<br>
From what I see in these two files: <a href="https://github.com/pytorch/text/blob/master/torchtext/data/field.py" rel="noopener nofollow ugc">field.py</a> and <a href="https://github.com/pytorch/text/blob/master/torchtext/vocab.py" rel="noopener nofollow ugc">vocab.py</a> it’s based on the frequency.</p>
<pre><code>    for word, freq in words_and_frequencies:
        if freq &lt; min_freq or len(self.itos) == max_size:
            break
        self.itos.append(word)
</code></pre>
<p>The code above ensures <code>itos</code> handles the <code>max_size</code>.</p>

vintagedeek: <p>Thank you for pointing me to this code! Yes, looking at the code it looks like it is based on frequency since the words are sorted by frequency (from most frequent to least frequent) per the first few lines below.</p>
<pre><code>   # sort by frequency, then alphabetically
    words_and_frequencies = sorted(counter.items(), key=lambda tup: tup[0])
    words_and_frequencies.sort(key=lambda tup: tup[1], reverse=True)

    for word, freq in words_and_frequencies:
        if freq &lt; min_freq or len(self.itos) == max_size:
            break
        self.itos.append(word)
</code></pre>
<p><strong>Edit</strong>: I think my original understanding was slightly off. If you set <code>max_size=50000</code>, then your vocab size should be 50002 for your 50000 most frequently occurring unique terms plus <code>&lt;unk&gt;</code> + <code>&lt;pad&gt;</code>.</p>
<p>From <code>vocab.py</code>,</p>
<pre><code>if specials_first:
    self.itos = list(specials)
    # only extend max size if specials are prepended
    max_size = None if max_size is None else max_size + len(specials)</code></pre>
Policy Gradient in Pytorch

Jayson_Ng: <p>Version 1</p>
<pre><code class="lang-auto">y = episode_a.argmax(-1)   # episode_a is in shape [T, n_actions]
action_preds = self.net(ep_s)  # action_preds is logits before softmax
neg_log_like = self.loss_fn(action_preds, y) 
loss = torch.mean(r * neg_log_like)   # r is reward
</code></pre>
<p>Version 2</p>
<pre><code class="lang-auto">y = torch.tensor(episode_a, requires_grad=True)
action_preds = model(ep_s)
neg_log_like = -y * torch.log(action_preds)
loss = torch.sum(neg_log_like, 1).mean()
</code></pre>
<p>Versions 1 and 2 seem to have same loss value. The difference is, y does not require grad in version while it does in version 2. But it is like a supervised learning backprop operation, y should be not needing require_grad. I do not understand why version 1 cannot train up the policy but version 2 can?</p>

Jayson_Ng: <p><a class="mention" href="/u/smth">@smth</a> May I have some help from you on this problem?</p>

Jayson_Ng: <p>Sorry for opening this post. I have solved the problem. It is due to another issue that is not mentioned here.</p>
What is the computational cost of x.to_sparse() and x.to_dense()

Michael_Moran: <p>I’m working with some very high dimensionality sparse tensors and I’m having RAM trouble with my dataloader. I’ve solved this by storing the tensors in sparse format on the dataloader but throwing a .to_dense() into the collate function. I just wanted to know if the .to_dense() operator has a high cpu cost associated with it or if its essentially “free”.</p>
Fine-tuning model with multiple losses

Anakin: <p>Hello,</p>
<p>suppose that a model consists of multiple components, i.e, object recognition model. A pseudo-model might look like es follows:</p>
<pre><code class="lang-auto">import torch

class FooModel(torch.nn.Module):
    def __init__(self):
        super(FooModel, self).__init__()
        
        # feature extraction
        self.backbone = ResNet101()
        
        # region proposal
        self.rpn = RegionProposalNetwork()
        
        # object detection
        self.localization = LocalizationModel()
        
        # object classification
        self.classifier = Classifier()
    
    def forward(self, inputs, targets)
        
        feats = self.backbone(inputs)
        
        rpn_loss_cls, rpn_loss_box, proposals = self.rpn(feats)
        
        roi_loss, box_feats = self.localization(feats, proposals)
        
        cls_loss = self.Classifier(box_feats, targets)

        return rpn_loss_cls, rpn_loss_box, roi_loss, cls_loss
</code></pre>
<p>Now I use <code>param.requires_grad = False</code> to freeze some layers in order to fine-tuning parameters. In this case, the gradients of those frozen layers are not calculated anymore. However, the losses are continuously calculated. A dummy fine-tuning strategy is defined below:</p>
<pre><code class="lang-auto">def FineTune(model, data_loader):
    for epoch in range(max_epoch):
        if epoch &lt; fine_tune_backbone:
            # freeze other models
            self.rpn.freeze_for_tune()
            self.localization.freeze_for_tune()
            self.classifier.freeze_for_tune()

            # unfreeze for fine-tuning
            self.backbone.unfreeze_me()
       elif epoch &gt; fine_tune_backbone and epoch &lt; fine_tune_rpn:
             self.rpn.unfreeze_me()
       elif epoch &gt; fine_tune_rpn and eopch &lt; fine_tune_roi:
             self.roi.unfreeze_me()

             self.localization.unfreeze_me()
      elif epoch &gt; fine_tune_roi and eopch &lt; fine_tune_cls:
            self.backbone.freeze_for_tune()
            self.rpn.freeze_for_tune()
            self.localization.freeze_for_tune()

            self.classifier.unfreeze_me()
      else:
            # training all together without backbone
            self.rpn.unfreeze_me()
            self.localization.unfreeze_me()

      inputs, targets = data_loader.get_batch()
      rpn_loss_cls, rpn_loss_box, roi_loss, cls_loss = model.forward(inputs, targets)
      ....
</code></pre>
<p>The question here is: how to fine-tune each part of this model. Am I doing it correctly or is there some canonical approach that PyTorch provides? If you have a look at some papers, there wrote just like <code>we fine-tuned xxx model after y-th epochs</code>. How does the implementation of fine-tuning look like in general? If some components got fixed, should I also change this submodel in inference mode (like, evaluation). Thanks for any inputs.</p>
Using data subsets

snowe: <p>Hi everyone <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>I have a question on how exactly <code>torch.utils.data.Subset()</code> works. From my understanding, it will return a subset of the data that consists of the same number of classes but only a subset of datapoints for each class. For example, <code>data = torch.utils.data.Subset(trainset, range(0, len(trainset), 2))</code> would give me half of the entire data with half of the datapoints for each class, right?</p>
<p>If so, is there an easy way to use a subset of the classes as well? In other words, a way to only take half of the classes of a dataset and maybe only half of the datapoints of each of those classes?</p>
<p>Any help is very much appreciated!</p>
<p>All the best,<br>
snowe</p>

Sobir_Bobiev: <p>Hi,<br>
Here is one way to do it:</p>
<pre><code class="lang-python">from torchvision.datasets import CIFAR10
trainset = CIFAR10('./data', train=True, download=True)

# select classes you want to include in your subset
classes = torch.tensor([0, 1, 2, 3, 4])

# get indices that correspond to one of the selected classes
indices = (torch.tensor(trainset.targets)[..., None] == classes).any(-1).nonzero(as_tuple=True)[0]

# subset the dataset
data = torch.utils.data.Subset(trainset, indices)

# or maybe half of the datapoints of each of those classes
data2 = torch.utils.data.Subset(trainset, indices[::2])

# a random half of the datapoints of each of those classes
total = len(data)
data3 = torch.utils.data.random_split(data, [total//2, total-total//2])[0]

</code></pre>

snowe: <p>Thank you <a class="mention" href="/u/sobir_bobiev">@Sobir_Bobiev</a>! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
How do I map class specific dataset?

TanmDL: <p>Let say, I have a mnist dataset with two classes (0 and 1). I want to create a dictionary which looks like dic= { ‘0’: all the zero sample images, ‘1’: all the one sample images}.</p>

TanmDL: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> please look into this problem.</p>
Projector tab is blank in Tensorboard (no checkpoint was found)

theairbend3r: <p>I’m trying to visualize embeddings per epoch. But the Projector page shows this -</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/9/89ddace5e21895e897e3ea18caf52eba7cb9f9f1.png" data-download-href="https://discuss.pytorch.org/uploads/default/89ddace5e21895e897e3ea18caf52eba7cb9f9f1" title="Screenshot from 2020-09-02 17-22-36"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/9/89ddace5e21895e897e3ea18caf52eba7cb9f9f1_2_690x342.png" alt="Screenshot from 2020-09-02 17-22-36" data-base62-sha1="jFClKlZrTcLPcTuBTbG2dHn8TVT" width="690" height="342" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/9/89ddace5e21895e897e3ea18caf52eba7cb9f9f1_2_690x342.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/9/89ddace5e21895e897e3ea18caf52eba7cb9f9f1_2_1035x513.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/9/89ddace5e21895e897e3ea18caf52eba7cb9f9f1_2_1380x684.png 2x" data-dominant-color="C0B7B4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2020-09-02 17-22-36</span><span class="informations">1920×954 186 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The text is as follows -</p>
<pre><code class="lang-auto">No checkpoint was found.

Probable causes:

    No checkpoint has been saved yet. Please refresh the page periodically.
    You are not saving any checkpoint. To save your model, create a tf.train.Saver and save your model periodically by calling saver.save(session, LOG_DIR/model.ckpt, step).

If you’re new to using TensorBoard, and want to find out how to add data and set up your event files, check out the README and perhaps the TensorBoard tutorial.

If you think TensorBoard is configured properly, please see the section of the README devoted to missing data problems and consider filing an issue on GitHub.
</code></pre>
<p>Graphs/Scalar/Histogram/Images all work fine. Only embedding does not.</p>
<p>To simplify things, I use the following to generate an embedding inside my training loop; still did not work.</p>
<pre><code class="lang-auto">for e in epochs:
    .
    .
    writer.add_embedding(torch.randn(100, 5), global_step=e)
</code></pre>
<p>What am I missing here?</p>

JisongXie: <p>Restart the tensorboard, and you will see it, as that projector loading is not dynamic. It will only show the embedding writed into logdir before you start tensorboard cli.</p>
Should I set feature extractor to eval mode when fine-tuning?

Sobir_Bobiev: <p>Hi,<br>
I am fine-tuning an ImageNet pretrained network on my own dataset. I have replaced the last fully-connected layer with the one that matches the number of classes in my dataset. And I have frozen the rest of the layers, which I consider as feature extractor. Now, I have two questions, that I think are somehow related to each other:</p>
<ol>
<li>The feature extractor contains some BatchNorm2D layers. So, should I set it to eval mode during training?</li>
<li>How to normalize the input? Using my dataset statistics or ImageNet’s statistics (mean=[0.485, 0.456, 0.406] and std=[0.229, 0.224, 0.225])?</li>
</ol>
<p>Thanks in advance.</p>
Unstability in ddpg

Antonio_Niro: <p>Hi, I’m trying to implement ddpg from scratch, following Lillicrap’s original paper (<a href="https://arxiv.org/pdf/1509.02971.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/1509.02971.pdf</a>). I’m having problems with the stability of the learning, and I really don’t know how to solve this problem. I tried different values for tau (parameter for target nets soft update), learning rate, batch size and dimension of actor and critic networks. I tried the algorithm on different mujoco envs too, and the results are almost the same you can see in the picture. Da you have any suggestion? Thanks in advance. <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/e/2efeb7e06e8bd747e51e884977c2c20374b9ddeb.png" data-download-href="https://discuss.pytorch.org/uploads/default/2efeb7e06e8bd747e51e884977c2c20374b9ddeb" title="obs_normalization"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/e/2efeb7e06e8bd747e51e884977c2c20374b9ddeb_2_690x364.png" alt="obs_normalization" data-base62-sha1="6HJH5DcLPQkCnW1gIda4HOna2WT" width="690" height="364" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/e/2efeb7e06e8bd747e51e884977c2c20374b9ddeb_2_690x364.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/e/2efeb7e06e8bd747e51e884977c2c20374b9ddeb_2_1035x546.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/e/2efeb7e06e8bd747e51e884977c2c20374b9ddeb_2_1380x728.png 2x" data-dominant-color="ECF2F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">obs_normalization</span><span class="informations">1848×975 187 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Resuming RandomSampler from an intermediate sample in a deterministic way

latorrefabian: <p>I would like to know if we can resume the iteration over a RandomSampler, from a given point in a deterministic way, i.e., stop execution in the middle of the iteration of a torch.utils.data.RandomSampler object, then saving RNG state, and later load RNG state in a way that the execution is equivalent to the original program, had we not stopped the execution. The following code sample summarizes the issue</p>
<pre><code class="lang-python">import torch
from torch.utils.data import TensorDataset, RandomSampler

print(torch.__version__)  # '1.6.0'
torch.manual_seed(0)
data = TensorDataset(torch.arange(10))  # [0, ..., 9]

# we will sample two random numbers with replacement
rand_samp = RandomSampler(data, replacement=True, num_samples=2)

state_0 = torch.get_rng_state() # save initial random state
for x in rand_samp:  # we print the two numbers
    print(x)  # 4; 9

torch.set_rng_state(state_0)  # go back to previous state

for x in rand_samp:  # print the first number and stop the sampler
    print(x)   # 4
    state_1 = torch.get_rng_state()  # try to save state after the first batch
    break

# script stops, we try to resume from the next batch

torch.set_rng_state(state_1)
for x in rand_samp:
    print(x)  # prints 3, but expected 9
    break
</code></pre>
<p>I think the way RandomSampler currently works, is that it prefetches all samples and then starts yielding one at a time, effectively loosing the ability to stop execution during the for loop and resuming later. One workaround is to create a RandomSampler with n_samples=1 at each iteration, that returns one sample at a time. In that way we can effectively save the RNG state of the underlying RandomSampler, but this is a bit cumbersome and probably inefficient.</p>
<p>I wanted to know if there is some better workaround, other than modifying the code of the RandomSampler class itself. The method to modify would be the <strong>iter</strong> method in line 113 of <a href="https://github.com/pytorch/pytorch/blob/master/torch/utils/data/sampler.py" class="inline-onebox" rel="noopener nofollow ugc">pytorch/sampler.py at master · pytorch/pytorch · GitHub</a></p>
<p>thanks for any help / feedback</p>
Is it possbile using a single gpu to predict multi images in parallelization？

wei.gao: <p>Hi guys,</p>
<p>in order to saving the inference time, I expect to use one gpu card to predict multi images concurrently,  could someone give me some suggestion？many thanks.</p>
<p>By the way, I have  tried to used more than one batch to predict, it will be ok. However, I want to use multiprocessing to predict multi images in parallel.</p>
<p>Kind regards</p>

ZdsAlpha: <p>Using multiple batches is actually the best you can get. You can also use multiple threads but make sure to use locks when using model (to keep memory requirement same). Using multithread will make sure GPU is being used even when other threads are loading image or doing pre/post processing.</p>
Noop flag device error with apex optimizers

yohann84L: <p>I’m getting the following <code>RuntimeError</code> using apex optimizer <code>FusedSGD</code>, (but I got it for all apex optimizers). I have not idea what it means, I checked, inputs, targets, loss, weights, all seems to be on the same cuda device.<br>
Any idea ? Should I open a bug issue on apex github ?</p>
<p>I’m using EfficientDet model by Ross Wightman (<a href="https://github.com/rwightman/efficientdet-pytorch" rel="noopener nofollow ugc">efficientdet-pytorch</a>)</p>
<pre><code class="lang-auto">...
...
  File "mylib/apps/training/lib/models/detection/effdet/engine.py", line 96, in train_one_epoch
    self.scaler.step(self.optimizer)
  File "mylib/torchenv/lib/python3.6/site-packages/torch/cuda/amp/grad_scaler.py", line 321, in step
    retval = optimizer.step(*args, **kwargs)
  File "mylib/torchenv/lib/python3.6/site-packages/apex/optimizers/fused_sgd.py", line 222, in step
    1.0/self.most_recent_scale)
  File "mylib/torchenv/lib/python3.6/site-packages/apex/multi_tensor_apply/multi_tensor_apply.py", line 30, in __call__
    *args)
RuntimeError: expected noop flag to be on the same device as tensors
</code></pre>
<p>Here’s my code to do the forward/backward step:</p>
<pre><code class="lang-auto">for images, targets in progress_bar(self.train_loader, parent=mb):
    targets = {k: v.to(self.device) for k, v in targets.items()}

    with torch.cuda.amp.autocast():
        loss_dict = model(images, targets)

    # Get loss values from dict
    loss = loss_dict["loss"]

    # Scales the loss, and calls backward()
    # to create scaled gradients
    scaler.scale(loss).backward()
    # Unscales gradients and calls
    # or skips optimizer.step()
    scaler.step(optimizer)
    # Updates the scale for next iteration
    optimizer.zero_grad()
    scaler.update()
</code></pre>
<p>and scaler is just <code>scaler = torch.cuda.amp.GradScaler()</code></p>

ptrblck: <p>Mixing native amp (via <code>torch.cuda.amp</code>) and <code>apex/amp</code> might yield these issues and we recommend to stick to the native implementation. <code>multi_tensor_apply</code> is already implemented in upstream PyTorch and more optimizers are being worked on to enable it.</p>
How to compute std of predcited tensors?

Alpay_Tekin: <p>I have an undirected graph data where G = (V, E). I implemented graph convolution to calculate node embedding. For each fold, I have 38 testing samples. My model computes node embedding and transforms them into adjacency matrix format. My question how can I compute the standard deviation of these 38 predicted adjacency matrices in each fold? Should I compute the standard deviation from node embedding? If so, what is the true way to calculate?</p>
What data format should I choose?

yonatansc97: <p>Hello,</p>
<p>I am writing my own pipeline, which sequentially generates data samples that consist of an <code>np.array</code> (with 3 dimensions ~ (100x100x15)), and a metadata <code>dict</code> (containing booleans, floats, integers and etc.). A single data sample with its metadata weighs ~600kb. That means that 10k samples are 6Gb in space. I am o.k with that during the saving of the pipeline, but during training of the model I don’t want to load 6Gb every time.<br>
Is there a good format that would allow me to save files containign 6Gb of information, but during reading not load it all at once?<br>
I know of <code>np.memmap</code>, but it would only work with pure tensors, and not with the metadata. And since I don’t want to separate images from their metadata, this is not a good option for me.</p>
<p>Thanks in advance,<br>
Jonathan</p>

hrushi: <p>I use hdf5 file format to store/load large datasets and h5py, the python library to deal with hdf5 files. It might be dificult to deal with it at start, but gets much better with experience</p>

yonatansc97: <p>Thanks for the reply!<br>
One question though: it says in the docs that " <strong>Groups work like dictionaries, and datasets work like NumPy arrays</strong>". In my case, the dataset should contain dictionaries (with an image and metadata keys). However, most examples I see have only integers/floats in them. Will hdf5 handle this well?</p>

hrushi: <p>I’m not really sure whether it can store metadata, but what I generally do is create two numpy arrays, one for images and other for labels. If I want the label for i-th image, I simply do label[i].</p>

yonatansc97: <p>For the sake of future readers, after learning about h5s I think this is the right solution. There are solutions for different dtypes, and it’s called “tables” <a class="mention" href="/u/hrushi">@hrushi</a></p>

hrushi: <p>I use these two <a href="https://github.com/pmusau17/ComputerVision/tree/master/Practioner_bundle/input_output" rel="noopener nofollow ugc">Modules</a> to write/retrieve datasets into/from hdf5 format. Maybe these might help you.</p>
PyTorch inference with multiple images

tominator: <p>I wrote an API to serve my PyTorch Object Detection Model. I followed the <a href="https://pytorch.org/tutorials/intermediate/flask_rest_api_tutorial.html" rel="noopener nofollow ugc">Tutorial</a> provided on the PyTorch website. Th aim is to send the API an image and to get back the result (classes, bounding boxes, etc.). For the prediction of only one image at time it works like a charm.</p>
<p>Now I want to extend it. The aim is to send multiple images to the API and to get back the results (in my case 5 images). I want to get the results as fast as possible. In a <a href="https://www.tensorflow.org/tfx/serving/performance#batch_size" rel="noopener nofollow ugc">Tensorflow Tutorial</a> I read that batching is a way to increase the performance. Is it better to do the predictions sequentially or to return the predictions for multiple images at once (batching)? What is the better and faster method? I run the inference on a GPU.</p>

1chimaruGin: <p>In my case, I used python <code>concurrent.futures.ProcessPoolExecutor</code> and <code>concurrent.futures.ThreadPoolExecutor</code> to inference 4 images at once.</p>
<p>But, this <a href="https://discuss.pytorch.org/t/how-to-load-images-for-inference-in-batch/33329/2">one</a> might help you.</p>

tominator: <p><a class="mention" href="/u/1chimarugin">@1chimaruGin</a> thanks for your reply. I will try your first suggestion. I also found the second entry, but it didn’t really help to reduce the inference time.</p>

1chimaruGin: <aside class="quote no-group" data-username="tominator" data-post="3" data-topic="109022">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/bb73d2/48.png" class="avatar"> tominator:</div>
<blockquote>
<p>to reduce the inference time.</p>
</blockquote>
</aside>
<p>Then, use TensorRT. It might help you a lot.</p>