akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Efficiency when pushing layers onto GPU after initialization

username: Byungheon_Jeong
cooked: <p>I was profiling the power draw of my GPU when I noticed that after the first time a layer is sent to the GPU using to(device= cuda_device), the speed at which the layers are sent to the GPU decreases by three orders of magnitude:</p>
<pre><code class="lang-auto">import timeit
import torch
from torch import nn

cuda = torch.device('cuda')

#1 Layer
def test():
      nn.ConvTranspose2d(in_channels=10, out_channels=32, kernel_size=(4,4),stride=2, padding=0).to(device=cuda)

#2 Layer
def test2():
      nn.ConvTranspose2d(in_channels=32, out_channels=32, kernel_size=(6,6),stride=2, padding=0).to(device=cuda)

# First time layer #1 is sent to the gpu, the time is at 3.6 secs
&gt;&gt;&gt; print(timeit.timeit(test, number=1))
3.634781403990928

# Second time the same (#1 Layer) layer is sent to the gpu, the time is at 0.0021 secs
&gt;&gt;&gt; print(timeit.timeit(test, number=1))
0.0021496260014828295

# This time, a different layer (#2 Layer) with larger input channel and larger kernels is sent to the gpu, the time is at 0.0027 secs
&gt;&gt;&gt; print(timeit.timeit(test2, number=1))
0.0026589079934637994

#After restarting the python interpreter and re-import everything, the same layer takes 3 orders of magnitude longer to send
&gt;&gt;&gt;exit()
&lt;Re-import all and define the test2 function&gt;
&gt;&gt;&gt;print(timeit.timeit(test2, number=1))
4.171760909986915
</code></pre>
<p>I have a feeling that this is clearly a front end cost for cuda initialization that pytorch establishes at the first to(gpu) call, but what exact reason behind this efficiency?</p>

username: ptrblck
cooked: <p>The first CUDA operation will create the CUDA context, which contains the the PyTorch kernels, cudnn, NCCL, CUDA libs etc., so it’ll take some time to load these libraries.</p>
<p>Also note, that CUDA operations are executed asynchronously, so if you want to profile the data transfer from host to device or any other CUDA op, you would have to synchronize the code before starting and stopping the timer via <code>torch.cuda.synchronize()</code>.</p>

username: Byungheon_Jeong
cooked: <p>Thank you! That was extremely helpful</p>
Word2Vec/GloVe Pretrained embeddings of custom dimensionality

username: Megh_Bhalerao
cooked: <p>I want to get the vector embeddings of a <strong>custom dimension</strong> using some word embedding models such as word2vec or GloVe on PyTorch?<br>
For example:</p>
<pre><code class="lang-auto">word = "cat"
output (of my desired size say 1024 dimension vector) = some_embedding_function(word)
</code></pre>
<p>Is there any way to do this on pytorch using some pre-trained models available?</p>
<p>Thanks!</p>

username: barthelemymp
cooked: <p>Hello,</p>
<p>If you use a pretrained embedding, the output dimension is then fixed by it. If you want to decrease the dim, I guess you could try a pca … but I’m not really sure it is a good idea.</p>

username: Megh_Bhalerao
cooked: <p>Hi <a class="mention" href="/u/barthelemymp">@barthelemymp</a>,<br>
Thanks for your reply. Is there any documentation on PyTorch that you can point me to?<br>
Thanks again</p>
Analyse own images in MNIST format

username: cruu
cooked: <p>Hello,<br>
i wrote a method to classify my own created images in MNIST format.<br>
Unfortunately it doesn’t return right results.</p>
<p>After training the NN, the implemented test method gives:<br>
“Test set: Average loss: 0.0253, Accuracy: 9922/10000 (99%)” so the NN should be able to classify my images pretty well.</p>
<p>Here the code:</p>
<pre><code class="lang-auto">def analyse(sample):
    use_cuda = torch.cuda.is_available()
    hardware = torch.device("cuda" if use_cuda else "cpu")
    model = Net().to(hardware)
    transform = transforms.Compose([
        transforms.ToTensor(),
        transforms.Normalize((0.1307,), (0.3081,))
    ])

    model.eval()
    image = Image.open(sample)
    image_tensor = transform(image)
    image_tensor_array = image_tensor.unsqueeze(0)
    # pil_image = transforms.ToPILImage(mode='L')(img_tensor)
    with torch.no_grad():
        data = Variable(image_tensor_array.cuda())
    # plt.imshow(pil_image)
    # plt.show()
    out = model(data)
    print(out.data.max(1, keepdim=True)[1]) 
    return str(out.data.max(1, keepdim=True)[1]) + "\n"
</code></pre>
<p>To avoid probably having the wrong format i tested my method with a original sample from MNIST.<br>
It seems to give random results.</p>
<p>thanks for all help…<br>
Dear</p>

username: ptrblck
cooked: <p>Could you explain your use case a bit more?<br>
If I understand the problem correctly, you’ve trained a model on MNIST and get a 99% accuracy.<br>
Now you would like to retrain the model on another dataset, with single-channel image tensors having a spatial size of 28x28, but the accuracy is bad?</p>
<p>PS: <code>Variable</code>s are deprecated since PyTorch <code>0.4</code> so you can use tensors now. You shouldn’t use the <code>.data</code> attribute, as it might yield unwanted side effects.</p>
Sampler missing in torchtext

username: Shanto_Islam
cooked: <p>I am currently working on sentiment analysis where the labels are unbalanced. I’ve found the similar <a href="https://discuss.pytorch.org/t/how-to-handle-imbalanced-classes/11264/5">solution</a> for Dataloader but haven’t found anything for torchtext. My torchtext code looks quite like this:</p>
<pre><code class="lang-auto">tokenize = lambda x: x.split()

TEXT = Field(sequential=True, tokenize=tokenize, lower=True, unk_token = None)
LABEL = Field(sequential=False, use_vocab=True, unk_token = None)

training, validation = TabularDataset.splits(
                        path = "./",
                        train = "training.csv", validation = "validation.csv",
                        format = "csv",
                        skip_header = True,
                        fields=[('text', TEXT), ('labels', LABEL)])

TEXT.build_vocab(training, max_size = None)
LABEL.build_vocab(training)

train_iter, val_iter, _ = data.BucketIterator.splits((
                    training, validation, _),
                    sort_key=lambda x: len(x.text),
                    batch_size = BATCH_SIZE, 
                    device = device))
</code></pre>
<p>The <em>sampler</em> argument seems to be missing in torchtext documentation.</p>
<p>Could anybody please help me on this matter?</p>
<p>Thanks in advance.</p>
How to differentiate weighted parameter updates?

username: benf.stokes
cooked: <p>Suppose we have a weighted loss (theta are the networks parameters):</p>
<p>o = model(data; theta)<br>
L = wi*F.cross_entropy(o, target, reduction=“none”),</p>
<p>where the wi are the result of another network (model_w). Assume we’re using SGD to update the parameters in model so that:<br>
theta’(wi) = theta-lr x wi x grad(F.cross_entropy(data, target, reduction=“none”))</p>
<p>If the weight network has a loss that’s of the form:</p>
<p>L_w = loss(data,target,model(data; theta’(wi)))</p>
<p>is there an easy way to backprop for the weight parameters? Or do I need to implement the backpropagation myself?</p>

username: albanD
cooked: <p>Hi,</p>
<p>The issue is that you want to backprop through the optmizer step?<br>
Well the issue is that the optimizers in pytorch are not differentiable <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"> So I would recommend using a library like <a href="https://github.com/facebookresearch/higher/">higher</a> that is built do differentiate through gradient updates!</p>

username: benf.stokes
cooked: <p>Something like that, yes <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> I see, I’ll take a look at higher, thanks for the recommendation!</p>
Inference Multiple Models on a Single GPU

username: Vival
cooked: <p>I am trying to build a system and i need to do inference on 60 segmentation models at same time ( Same models but different inputs). I wonder if this is possible to do on pytorch?<br>
I am not sure about what kinda system i need to use for this, i am planning to use 4X RTX8000 and if it is not enough i can use two system with 4X RTX8000 each or with a better gpu.</p>
<p>Would i lose too much performance because of using multiple models? How many models i can put on a gpu, what the performance of it depends on? Is it just vRAM of gpu or processing speed? Sorry for asking such trivial questions but i really couldn’t come up with an answer by searching.</p>
<p>You can assume that i am going to use Yolact, <a href="https://github.com/dbolya/yolact" rel="nofollow noopener">https://github.com/dbolya/yolact</a><br>
I would be very happy if you can help me, thank you so much and sorry for my English/grammar.</p>

username: iffiX
cooked: <p>The simplest and probably the most efficient method whould be concatenate your samples in dimension 0 (i.e. the batch dimension).<br>
If that is too much for one gpu, then wrap your model in <code>DistributedDataParallel</code> and let it handle the batched data.<br>
Do not use multiple models unless they hold different parameters. It’s unecessary.</p>

username: Vival
cooked: <p>It makes sense. I was planning to use 30 robotic arms, 2 model for each. By doing like that i guess every robotic arm will get info about what to do on same time. I think i can use that, i 've read about running multiple models on tensorflow is possible so that is why i wanted to know if it is possible on pytorch, thanks for answer again.</p>
Scaler.get_scale() becomes zeros and loss is still nan under autocast

username: Saurabh_Kataria
cooked: <p>I have been trying to use autocast. My loss is nan and therefore <code>scaler.get_scale()</code> keeps reducing after every batch from 32768 until it becomes zero, so i stop the training. Traceback shows “RuntimeError: Function ‘PowBackward0’ returned nan values in its 0th output.” I have tried to increase epsilon value in the line of code which gives this error. I have also tried to increase the epsilon of Adam optimizer that I’m using to 1e-4.</p>

username: ptrblck
cooked: <p>If your model outputs NaN values, the training is generally broken and I don’t know if the model can somehow recover from it. Additionally the <code>scaler</code> will reduce the loss scaling value, as the backward pass creates NaN gradients, and it thus assumes that the scaling factor is too high.</p>
<p>I would recommend to remove automatic mixed precision for now and make sure to fix the NaN issue.<br>
Based on the error message, you might pass a negative values to a <code>pow</code> method, which would calculate a root?</p>
Understanding target.eq(pred).sum().template item<int64_t>();

username: yeedTorch
cooked: <p>I am just starting out with libtorch and pytorch as well, so I apologize if I haven’t searched enough for an answer before posting.</p>
<p>Looking at some examplary code found online and trying that on my own machine I have stumbled upon this expression: target.eq(pred).sum().template item&lt;int64_t&gt;();</p>
<p>I have never seen a .template item&lt;&gt;() like expression in C++ before, maybe I am just too unexperienced  with templates in C++ but I don’t understand what that is. Is it some form of template metaprogramming that I am not familiar with? Or something libtorch specific?</p>

username: ptrblck
cooked: <p>The <code>template</code> keyword is to disambiguate the actual call, as otherwise the expression might be understood by the compiler as <code>item less-than int64_t greater-than ()</code>.<br>
<a href="https://stackoverflow.com/a/613132">This post</a> goes into detail and can explain it way better than I can. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
How to set random seed when it is in distributed training?

username: sunshk1227
cooked: <p>Hi everyone!</p>
<p>Now I am training a model using <code>torch.distributed</code>, but I am not sure how to set the random seeds. For example, this is my current code:</p>
<pre><code class="lang-python">def main():
    np.random.seed(args.seed)
    torch.manual_seed(args.seed)
    torch.cuda.manual_seed(args.seed)

    cudnn.enabled = True
    cudnn.benchmark = True
    cudnn.deterministic = True 

    mp.spawn(main_worker, nprocs=args.ngpus, args=(args,))
</code></pre>
<p>And should I move the</p>
<pre><code class="lang-python">    np.random.seed(args.seed)
    torch.manual_seed(args.seed)
    torch.cuda.manual_seed(args.seed)

    cudnn.enabled = True
    cudnn.benchmark = True
    cudnn.deterministic = True 
</code></pre>
<p>into the function main_worker() to make sure every process has the correct seed and cudnn settings? By the way, I have tried this and this behavior will make the training process 2 times slower, which really confused me.</p>
<p>Thank you very much for any help!</p>

username: ptrblck
cooked: <p>Each process should execute the seeding code.<br>
The slowdown might come from e.g. <code>cudnn.deterministic = True</code>, as this will use the default algorithm, which might be slower than the others.<br>
Also, <code>cudnn.benchmark = True</code> won’t have any effect, if you set <code>cudnn.deterministic = True</code>.</p>

username: sunshk1227
cooked: <p>Thank you very much! I get it!</p>
How to remove specific datapoints from dataset during training?

username: Maria_Fernanda_De_La
cooked: <p>Hello, I would like to remove specific datapoints during training, I would need to keep track of the ‘global’ index of a particular image, remove and have a new loader with the new set (after removal) set. What is the best way to do this?</p>

username: ptrblck
cooked: <p>I think one easy approach would be to</p>
<ul>
<li>store the indices of all data samples, which should be removed in a <code>list</code> during training</li>
<li>remove these indices from the complete list of indices to create the “valid indices”</li>
<li>wrap the <code>dataset</code> in a <code>Subset</code> with the new “valid” indices,</li>
<li>and recreate the <code>DataLoader</code> after the epoch.</li>
</ul>
How much input size of pretrained model affect classification model

username: mhxin
cooked: <p>when i use pre-trained model(like mobilenetv2) to train my classification model, at first i set the input size of my classicication model as $64\times 128$, the effects was not good， later i try to change my input size to $128 \times 256$, i retrain my model, and get a better result.</p>
<p>is that means when i use pre-trained model from imagenet, The input size needs to be as close as possible to the original input(224 \times 224) of the pre-trained model。 and why it matters to a classification model.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="mhxin" data-post="1" data-topic="83446">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mhxin/48/22307_2.png" class="avatar"> mhxin:</div>
<blockquote>
<p>is that means when i use pre-trained model from imagenet, The input size needs to be as close as possible to the original input(224 \times 224) of the pre-trained model。 and why it matters to a classification model.</p>
</blockquote>
</aside>
<p>Generally, the spatial resolution should match the original input resolution, as the conv kernels have been trained to extract useful features with this particular setup.<br>
E.g. edge detector filters, which were trained on a particular resolution might not detect edges, if you zoom in/out of the image.</p>
<p>I guess the spatial resolution must not necessarily be <code>224x224</code>, if the “pixel spacing” would stay the same. I.e. if the edges and other features still use the same amount of pixels, the filters might still work fine, so you could run some experiments to verify it.</p>