akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to decrease gpu usage in F.grid_sample?

username: DRosemei
cooked: <p>I have a ground thruth tensor of [n_imgs, classes, H, W], and I expand it to [batch_size, n_imgs, classes, H, W] (expand does not repeat it in memory). Then for each img, I  got  [batch_size, classes, H, W] tensor, and I use it for the input of F.grid_sample. Forward computing works well while in backward, the gpu usage is around 76 GB. I try to compute it each bach while it costs too much time. So how to construct the input tensor to backward gradient fast?</p>
CrossEntropyLoss backpropagation

username: Sam-gege
cooked: <p>hi, according to the <a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html" rel="noopener nofollow ugc">doc</a>, when it says " This criterion combines <a href="https://pytorch.org/docs/stable/generated/torch.nn.LogSoftmax.html#torch.nn.LogSoftmax" rel="noopener nofollow ugc"><code>LogSoftmax</code></a> and <a href="https://pytorch.org/docs/stable/generated/torch.nn.NLLLoss.html#torch.nn.NLLLoss" rel="noopener nofollow ugc"><code>NLLLoss</code></a> in one single class." Does it mean to simply connect these two modules, i.e. connect the output of LogSoftmax to the input of NLLLoss?</p>
<p>I’d like to ask this because I learnt that when combining these two modules, the backpropagation may be simplified. For example, if the input is x1,x2, their softmax is s1, s2, and output is y1, y2, then dLoss/dx1=s1(y1+y2)-y1. y1+y2 is just 1 so it can be cancelled so we won’t need to construct a Jacobian because there’s no contribution from y2. However, when there’s weight, i.e. w1y1+w2y2, or ignore_index, i.e. 0*y1+y2, y2 can’t be cancelled out, so there’s no difference if we simply connect those two modules together? (not sure, my speculation).</p>
<p>I don’t understand C so I can’t confirm it, could someone help me on this? I’m a new learner and trying to implement those modules myself, so far I’ve implemented logsoftmax and NLLLoss, It would be easy if CrossEntropyLoss simply connect them together <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"> Thanks for any help.</p>

username: Sam-gege
cooked: <p>Oh never mind, they simply linked together. I thought they are not written in python as others.<br>
so in nn/functional.py it has:<br>
return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)</p>
A bug when I do inference on a saved model

username: HaythemBD
cooked: <p>Hello Guys, I’m having a small isuue ith a saved model. I trained the model on colab (GPU) then I download the model(pth format) and then I tried to use it locally and this error occured<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/f/5f9f1b5b196ea8818ddd6250696a47f017c6849f.png" data-download-href="https://discuss.pytorch.org/uploads/default/5f9f1b5b196ea8818ddd6250696a47f017c6849f" title="error_pytorch"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/f/5f9f1b5b196ea8818ddd6250696a47f017c6849f.png" alt="error_pytorch" data-base62-sha1="dDUfa6OqcAKQWtFz24Byd0uDtOD" width="690" height="440" data-dominant-color="F3F5F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">error_pytorch</span><span class="informations">1090×696 56.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Any help is much appreciated.</p>

username: ptrblck
cooked: <p>Saving the “entire” model via <code>torch.save(model)</code> and loading it afterwards might easily break, since you would have to restore the same file structure again and make sure the code is also unchanged.<br>
Based on the error it seems that your local setup diverges from this assumption, which could be caused by manual changes to (some) files or different library versions.<br>
The current implementation of <a href="https://github.com/pytorch/vision/blob/932ca5a39a84d6625dbe61d1b330aafa7b3cfa0d/torchvision/models/detection/anchor_utils.py"><code>AnchorGenerator</code></a> doesn’t provide a <code>_cache</code> attribute, so I’m unsure where this is coming from (maybe an older release did?).</p>
<p>Check <a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html">this tutorial</a> which explains how to save and load the <code>state_dict</code> instead.</p>
Multi-pair losses

username: jhp
cooked: <p>Hi I have a question about <code>calculating losses for multiple-pairs</code>.</p>
<p>As far as I know, Pytorch support calculating losses only between input and target.<br>
In my case, I have several embedding pairs (i.e A:(N,D),B:(N,D),C:(N,D)) and want to pull each other by using like L2 loss.<br>
Is there any internal function that can handle multiple pairs of inputs to calculate losses like the above case in Pytorch?</p>
<p>If not, what is the right way to pull multiple pairs of inputs?</p>
<p>I think there are two cases:</p>
<pre><code class="lang-auto">case1
input:A ,target:B
input:A ,target:C
input:B ,target:A
input:B ,target:C
input:C ,target:A
input:C ,target:B
</code></pre>
<pre><code class="lang-auto">case2
input:A ,target:B
input:A ,target:C
input:B ,target:C
</code></pre>
<p>cf) Reason why I think of two cases, is because I wonder whether the target embeddings also get gradients or not</p>

username: ptrblck
cooked: <p>I’m not sure I understand the use case completely, but you can manually calculate the loss using al (differentiable) PyTorch operations and don’t necessarily need to stick to the pre-defined loss functions defined in the <code>nn</code> namespace.</p>
How to free the pytorch transformers model from GPU memory

username: yanan_chen
cooked: <blockquote>
<pre><code>    from transformers import CTRLTokenizer, TFCTRLLMHeadModel
    tokenizer_ctrl = CTRLTokenizer.from_pretrained('ctrl', cache_dir='./cache', local_files_only=True)
    model_ctrl = TFCTRLLMHeadModel.from_pretrained('ctrl', cache_dir='./cache', local_files_only=True)
    gen_nlp  = pipeline("text-generation", model=model_ctrl, tokenizer=tokenizer_ctrl, device=args.gpu, return_full_text=False)
</code></pre>
</blockquote>
<p>Hello, I want to remove the model  <code>model_ctrl</code> from the GPU memory after using it, to free the memory it occupied.</p>
<p>What is the best way to to this ?<br>
Thanks.</p>

username: ptrblck
cooked: <p><code>del</code> all objects related to the model, i.e. the model itself and potentially optimizers, which could hole references to the parameters and if you want to clear the cached memory to allow other applications to use it, call <code>torch.cuda.empty_cache()</code>.</p>
Cross-entropy loss for sequence of elements

username: kenenbek
cooked: <p>I have a sequece labeling task.</p>
<p>So as input, I have a sequence of elements with shape <code>[batch_size, sequence_length]</code> and I need to assign a class for each element of a sequence.</p>
<p>And as a loss function, I use a <a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html" rel="noopener nofollow ugc">Cross-entropy</a>.</p>
<p>How should I correctly use it?<br>
My variable <code>target_predictions</code> has shape <code>[batch_size, sequence_length, number_of_classes]</code> and <code>target</code> has shape <code>[batch_size, sequence_length]</code><br>
Documentation says:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/f/3fa4d0bf54ae9dec2fead700fff9f61dc34cfade.png" data-download-href="https://discuss.pytorch.org/uploads/default/3fa4d0bf54ae9dec2fead700fff9f61dc34cfade" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/f/3fa4d0bf54ae9dec2fead700fff9f61dc34cfade_2_690x116.png" alt="image" data-base62-sha1="951c9Bf593wqBv8lOOJewSHKfJY" width="690" height="116" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/f/3fa4d0bf54ae9dec2fead700fff9f61dc34cfade_2_690x116.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/f/3fa4d0bf54ae9dec2fead700fff9f61dc34cfade_2_1035x174.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/f/3fa4d0bf54ae9dec2fead700fff9f61dc34cfade_2_1380x232.png 2x" data-dominant-color="F2F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1430×242 21.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I know if I use <code>CrossEntropyLoss(target_predictions.permute(0, 2, 1), target)</code>, everything will work fine. But I have concerns that in case of <code>k-dimensional loss</code>, torch will intepret my <code>sequence_length</code> as <code>d_1</code> as on screenshot.</p>
<p>How should I correctly do it?</p>

username: googlebot
cooked: <aside class="quote no-group" data-username="kenenbek" data-post="1" data-topic="133017">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kenenbek/48/19604_2.png" class="avatar"> kenenbek:</div>
<blockquote>
<p>torch will intepret my <code>sequence_length</code> as <code>d_1</code></p>
</blockquote>
</aside>
<p>and why is this wrong?</p>
Capturing the computation graph

username: lkp411
cooked: <p>Hello there,</p>
<p>Is there a way to capture a computation graph of an nn.Module other than torch.fx? The issue I am having with torch.fx is that when the forward function contains conditionals, static capture doesn’t seem to work. I am trying to obtain the computation graph that is identical to that which torch.fx enables us to obtain but also want it to work for forward functions that have conditionals in them. For instance, I can capture the computation graph of</p>
<blockquote>
<p>nn.TransformerEncoderLayer</p>
</blockquote>
<p>but not</p>
<blockquote>
<p>nn.Transformer</p>
</blockquote>
<p>using torch.fx.</p>
Extract Features using pytorch

username: samm
cooked: <p>excuse me I’m confused about questions . I need to extract the features before the classification layer as i don’t need to classify the image but need the features that help to classify only</p>
<p>Suppose I have a classification model called MyModel.py</p>
<p>After removing the output layer.</p>
<p>1- Should I train the class of the model on my dataset ?</p>
<p>2- I should get checkpoint .pth after training . Will this file be the weight ?</p>
<p>3- if I will train the model after remove the output layer . Should I save tensor in</p>
<pre><code class="lang-auto">def forward (self,x)

x = self.forward_features(x)

#x = self.head(x) # here I will save the tensor ??

return x
</code></pre>
<p>Or just the checkpoint.pth that I will get enough</p>
<p>4- the model comes with a .pth file ( pre-trained file) … but there is one for the classification and I shouldn’t use it . Right</p>

username: JuanFMontesinos
cooked: <p>There are some models pretrained on imagenet which generalise very well even for unseen classes.<br>
You can trust that if you have no much clue about what u are doing.</p>
<p>If the model is not trained then yes, you should train it in your dataset.<br>
Anyway randomly-initlised CNNs still works better than random.<br>
The pth file is a compressed file, it contains what you save into it.<br>
Assuming you saved the weights of your model, then yes.<br>
You can do both, use the model on-the-fly if you have a good GPU and a small hard disk or save the tensor if you need it.<br>
That’s up to your necessities.<br>
Models usually have a last layer to compute class probabilities. If you train your model from scratch you can just delete it. If you use a pretrained model you need to keep it as you will have an error otherwise.</p>
Hyper-net cannot change the weights of the backbone-net using state_dict()

username: Laurent1
cooked: <p>Hi,</p>
<p>I want to use a hyper-net to change the weights of a backbone-net. The following snippet is not meaningful but illustrates in a concise way what I want to do:</p>
<pre><code class="lang-auto">
import sys
import torch
from torch import nn, optim

# our backbone-net
modelV = nn.Linear(1, 1, bias=False)
nn.init.zeros_(modelV.weight)
# our hyper-net
modelW = nn.Linear(1, 1, bias=False)
nn.init.constant_(modelW.weight, 0.1)

optimizer = optim.Adam(modelW.parameters(), lr=1e-3)
loss_func = nn.MSELoss()

# our data
# the hyper-net takes in z = 1 and must output 2 if trained
# the backbone-net takes in X and must output Y if trained with Y = 2 X
Z = torch.tensor(1.).reshape(-1,1)
X = torch.Tensor( [ -1. , 0 , 1., 2. ] ).reshape(-1,1)
y = torch.Tensor( [ -2., 0., 2., 4. ] ).reshape(-1,1)

for ite in range(10):
    hypernet_prediction = modelW(Z)
    # build a backbone-net
    backbone_state_dict = modelV.state_dict()
    backbone_state_dict["weight"] = hypernet_prediction[0].view(backbone_state_dict["weight"].size())
    modelV.load_state_dict(backbone_state_dict)
    # predict with backbone-net
    backbonet_prediction = modelV(X)
    # compute loss
    loss = loss_func(backbonet_prediction, y) 
    print("ite = {}, loss = {}".format(ite, loss.item()))
    # optimize
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
</code></pre>
<p>This way of changing weights is very elegant but unfortunately it does not work because as mentioned in other threads the loading of the state dict is not tracked as a differentiable op by autograd.</p>
<p>What alternatives are there ? (keeping in mind that the real models will be much much larger)<br>
thanks for any help</p>
Implement sequential stacked CNN + LSTM

username: Atefeh_Rashidi
cooked: <p>I am trying to combine CNN and LSTM for the textual data for purpose of classifying.</p>
<p>Could someone give me some examples of how to implement a CNN + LSTM structure in PyTorch? I know I should use word embeddings to convert text to vector. Then, these vectors should be sent to CNN and the output of CNN (features) should be fed into the LSTM network and then a sigmoid layer should be applied. But I have no idea how to implement this by PyTorch?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/a/7a4e095c139f9520ec5773201b65a851dc0b74f5.png" data-download-href="https://discuss.pytorch.org/uploads/default/7a4e095c139f9520ec5773201b65a851dc0b74f5" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/a/7a4e095c139f9520ec5773201b65a851dc0b74f5.png" alt="image" data-base62-sha1="hrXrwzaYBXVMHrfOfZ0gVnUOXnD" width="690" height="207" data-dominant-color="B2D1DC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">700×211 3.69 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Thanks!</p>
How to transpose nn.Embedding?

username: RylanSchaeffer
cooked: <p>I have a nn.Embedding layer. It maps integers to vectors of some dimension. We can think of it as a matrix times one-hot embedded inputs vectors i.e. <code>torch.matmul(E, x)</code>. Given a vector <code>y</code> of the output dimension of <code>E</code>, I want to compute a probability distribution over the input space i.e. <code>softmax(torch.matmul(E.T, y))</code>. How do I do this?</p>

username: Scott_Hoang
cooked: <p>why not just retrieve all the embedding manually then multiply by y?<br>
something like<br>
softmax(torch.matmul(E[torch.arange(E.num_embedding).long()], y))?</p>

username: RylanSchaeffer
cooked: <p>Would <code> softmax(torch.matmul(E.weight.T, y))?</code> still preserve backprop to the embedding matrix?</p>

username: Scott_Hoang
cooked: <p>oh yeah, that should works too.</p>
Multiplication of tensors

username: Marek_Balaz
cooked: <p>Hello, I would like to multiply tensor t1 = torch.tensor([1, 2, 3, 4, 5]) and t2 = torch.tensor([1, 2, 1]) in order to get result similar like from 1D kernel application: [1 * 1 + 2 * 2 + 3 * 1, 2 * 1 + 3 * 2 + 4 * 1, 3 * 1 + 4 * 2 + 5 * 1]. Can you give me advice, how to compute this case without any for loop?</p>
<p>Thank you. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: KFrank
cooked: <p>Hi Marek!</p>
<aside class="quote no-group" data-username="Marek_Balaz" data-post="1" data-topic="132981" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/marek_balaz/48/30834_2.png" class="avatar"> Marek_Balaz:</div>
<blockquote>
<p>how to compute this case without any for loop?</p>
</blockquote>
</aside>
<p>Use <a href="https://pytorch.org/docs/1.9.1/generated/torch.Tensor.unfold.html" rel="noopener nofollow ugc">unfold()</a> to generate the correct “slices” of <code>t1</code>:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.9.0'
&gt;&gt;&gt; t1 = torch.tensor([1, 2, 3, 4, 5])
&gt;&gt;&gt; t2 = torch.tensor([1, 2, 1])
&gt;&gt;&gt; t1.unfold (0, 3, 1) @ t2
tensor([ 8, 12, 16])
</code></pre>
<p>You could also use <a href="https://pytorch.org/docs/1.9.1/generated/torch.nn.functional.conv1d.html" rel="noopener nofollow ugc">conv1d()</a> if you <code>unsqueeze()</code> <code>t1</code> and <code>t2</code> to add<br>
the required singleton dimensions.</p>
<p>Best.</p>
<p>K. Frank</p>
Speech recognition and generation

username: VictorM
cooked: <p>Hello experts in DL and Pytorch,<br>
I have multiple mp3 files with a voice of mine (and corresponding txt files)<br>
How is it possible to train a Pytorch model, so it will make a speech-to-text generation of any text with my voice? Please give me hints/tips/ideas.</p>
<p>Thank you in advance.</p>

username: Hengineer327
cooked: <p>I’m also having this problem. Did you check out Micheal Phi’s youtube video: <a href="https://www.youtube.com/watch?v=YereI6Gn3bM&amp;t=5s" class="inline-onebox" rel="noopener nofollow ugc">I Built a Personal Speech Recognition System for my AI Assistant - YouTube</a></p>
<p>Comment on me if you figure this out, and if I figure it out, it will be the same for you.</p>
Transformers GPT2 too large for Google Colabs Pro?

username: Max_Unhold
cooked: <p>Hello,</p>
<p>I would like to benchmark several PyTorch transformer models on the</p>
<ul>
<li><code>AG_NEWS dataset (torchtext.datasets)</code></li>
<li>using google colabs pro with 15 GB GPU available memory.</li>
</ul>
<p>While ‘BERT’,‘DistilBert’,‘Electra’,…,‘XLM-RoBERTa’ and their tokenizers work without a problem, both ‘GPT 2’ and ‘XML’<br>
do not fit in the memory.</p>
<p>Since 15 GB seems to be a lot, I wonder wether</p>
<ul>
<li>it is a coding problem on my site or both models are just very large</li>
<li>there is a light-weight version of GPT and XML</li>
</ul>
<p>Kind regards</p>
Backpropagating losses

username: Rituraj_19_Dutta_100
cooked: <p>I have written a code where there are 3 losses basically segmentation loss, adversarial loss and discriminator loss  and the total loss is something like this:</p>
<blockquote>
<p>Ltotal = L seg + λ1 L adv + λ2 L disc</p>
</blockquote>
<p>In terms of code the final loss is:</p>
<pre><code class="lang-auto">        output = model(x)

        bce = nn.BCELoss().cuda(GPU)

        ## Pixel Wise Loss

        s_loss = bce(output,Variable(ground_truth).cuda(GPU))

        ## Measures Segmenter's ability to fool the discriminator

        adv_loss = bce(critic(fake_img),valid)
        

        ###### Train Discriminator ########

        ## Measures Discriminator's abilty to classify real from generated images

        real_loss = bce(critic(real_img),valid)
        fake_loss = bce(critic(fake_img),fake)
        d_loss = (real_loss + fake_loss) / 2

       

        loss = s_loss + lambda1 * adv_loss + lambda2 * d_loss 

        # training
        model.zero_grad()
        discriminator.zero_grad()

        loss.backward()
</code></pre>
<p>I just wanted to know if this is the correct way of interpreting the 3 losses together.</p>
<p>P.S: This is somewhat a modified DCGAN with a segmentation network inplace of the generator</p>

username: shivammehta007
cooked: <p>Hello,<br>
The implementation looks correct to me given the provided code.</p>
RuntimeError: CUDA out of memory with resizing the image

username: mathwseg
cooked: <p>i got this error</p>
<pre><code class="lang-auto">  File "fin.py", line 80, in extract_features
    output = model(tensor_on_device)
  File "/home/user/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/model.py", line 238, in forward
    x = self.forward_features(x)
  File "/model.py", line 229, in forward_features
    x = blk(x, H, W)
  File "/home/user/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/model.py", line 97, in forward
    x = x + self.drop_path(self.mlp(self.norm2(x)))
  File "/home/user/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/model.py", line 26, in forward
    x = self.fc1(x)
  File "/home/user/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/user/.local/lib/python3.6/site-packages/torch/nn/modules/linear.py", line 94, in forward
    return F.linear(input, self.weight, self.bias)
  File "/home/user/.local/lib/python3.6/site-packages/torch/nn/functional.py", line 1753, in linear
    return torch._C._nn.linear(input, weight, bias)
RuntimeError: CUDA out of memory. Tried to allocate 2.00 MiB (GPU 0; 5.80 GiB total capacity; 4.12 GiB already allocated; 4.25 MiB free; 4.13 GiB reserved in total by PyTorch)
</code></pre>
<p>the problem here in resize… every time I reduce resize … the code keeps working more … but stop and didn’t complete all images I have… I tried to set resize to 32 and reach at 2124 images only … How can i solve it ?</p>
<pre><code class="lang-auto">inference_transform = T.Compose(
      [

         T.Resize(64),

         T.ToTensor(),

         T.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]), 

      ]
    )
</code></pre>
<p>i set model work on Cuda and tried also CPU but got killed in the middle of run</p>

username: ptrblck
cooked: <p>I don’t think the issue is caused by <code>Resize</code>, but seems to be delayed for smaller shapes.</p>
<aside class="quote no-group" data-username="mathwseg" data-post="1" data-topic="132961">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mathwseg/48/39221_2.png" class="avatar"> mathwseg:</div>
<blockquote>
<p>the code keeps working more … but stop and didn’t complete all images I have</p>
</blockquote>
</aside>
<p>This indicates that your script is increasing the memory usage in each iteration, which could be caused by e.g. storing the model output or loss in a <code>list</code> without detaching them.<br>
Check the memory usage via <code>nvidia-smi</code> or <code>torch.cuda.memory_summar()</code> in each iteration and you would most likely see an increase in each step.</p>
CPU and GPU usage low during training

username: Louis1
cooked: <p>Hi,</p>
<p>I’m using pytorch on an aws machine which runs on linux. It has 32 CPUs and 1 GPU.</p>
<p>My data is stored on the SSD of the machine and is loaded during the training in the Dataset. I am using the DataLoader class to feed data to the model during the training. I notice that neither the GPU nor the CPU are saturated. I use <code>watch -n 0.5 nvidia-smi</code> to monitor GPU usage and <code>htop</code> to monitor CPU usage. GPU usage is around 30% average. The 32 CPUs are 100% used during the very beginning of the training (maybe first batch, only a few seconds) but then only 4 or 5 are used during the rest of the training.</p>
<p>If the GPU is the bottleneck then it should be around 100% all the time and if the CPUs were the bottleneck I would expected the same for all 32 of them.</p>
<p>How come CPU usage AND GPU usage are so low simultaneously ? How can I speed up training ?<br>
I have tried mixed precision and other stuff but my question is mostly about understanding why CPU and GPU usage are so low.</p>
<p>Here is my (train) dataloader :</p>
<pre><code class="lang-auto">DataLoader(dataset=visits_dataset, batch_size=256, collate_fn=pad_collate,
                              sampler=SubsetRandomSampler(train_indices), pin_memory=True, num_workers=32)
</code></pre>
<p>I would be very thankful for any help.</p>

username: ptrblck
cooked: <p>To get a clear answer you could profile the code e.g. via Nsight Systems (or the PyTorch profiler) and check the timeline to see which operations are executed.<br>
Based on your description, I would guess that the CPU might be busy with some (single-core) work.</p>

username: Louis1
cooked: <p>Thank you very much for your answer, I will profile the code.</p>
How to get the intermediate features from Alexnet

username: nikiguo93
cooked: <p>Hi, can anybody tell me how to use the forward hook function to get the intermediate features from Alexnet?</p>

username: ptrblck
cooked: <p><a href="https://discuss.pytorch.org/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/6">Here</a> is an example for a custom model and you would need to replace the layer names in order to use it with AlexNet.</p>
Slow forward inference for traced model on second infrence, data dependant or independant?

username: drydenwiebe
cooked: <p>Hello,</p>
<p>I have a traced model that runs very slowly on the second inference run on torch 1.7 and greater. From <a href="https://discuss.pytorch.org/t/slow-forward-on-traced-graph-on-cuda-2nd-iteration/118445">these</a> <a href="https://github.com/pytorch/pytorch/issues/50796" rel="noopener nofollow ugc">posts</a> and others, this seems to be the expected behavior.</p>
<p>I have one question about the exact speed of the jit/graph optimisation run (the second run). Say that my model can take in different sized tensors as data during the forward pass, say [batch_size, c, h, w] or and then [batch_size, c * 10, h, w] both being valid inputs. Would be it expected that two forward passes (including the second, longer one) would be faster with input [batch_size, c, h, w] than input sized [batch_size, c * 10, h, w]? My experiments show that the second run (that optimises the graph) is the same (slower) speed with both inputs. I just want to confirm that this is the expected behavior.</p>
<p>Thank you!</p>

username: ptrblck
cooked: <p>If I’m not mistaken, the default JIT backend uses up to 3 steps to optimize the graph for each new input shape. I.e. the new and larger batch would trigger the optimization again. The <code>nvfuser</code> backend would relax this slowdown a bit, as it’s more flexible when it comes to different input shapes (in case the generated kernels cannot be reused, they would be regenerated and optimized again).</p>
How to deploy a Text to Speech model on AWS

username: Hasan_Khan
cooked: <p>Hello,<br>
I’m trying to deploy my Trained Text to speech model using AWS lambda services but as I haven’t worked on the deployment part before so I’m having trouble to figure out the initial steps. My model consist of two trained models one is waveglow for inferencing and the other one is my trained tacotron2 model on a specific language. I know how to create a lambda function and trigger s3 bucket in which my model is saved but cant figure out how to define a lambda function. Please guide me with the process. Thanks in advance.</p>

username: dbish
cooked: <p>Hey, did you ever solve your problem? Which tutorial or options did you try? I’m curious if you tried using sagemaker for model hosting since you’re already on AWS, such as described here: <a href="https://sagemaker-examples.readthedocs.io/en/latest/frameworks/pytorch/get_started_mnist_deploy.html" class="inline-onebox" rel="noopener nofollow ugc">Deploy a Trained PyTorch Model — Amazon SageMaker Examples 1.0.0 documentation</a>.</p>
Trivial model on iOS gets EXC_BAD_ACCESS on forward call

username: beakin
cooked: <p>I think I’m missing something basic, so apologies if this is obvious.</p>
<p>Running pytorch 1.9.1; ios LibTorch-Lite 1.9.0</p>
<p>I’m trying to run a model in iOS under c++, and during the call to <code>forward()</code>, the <code>get_method("forward")</code> call works, and then the thread crashes on the <code>std::move(inputs)</code> call with a EXC_BAD_ACCESS error.</p>
<p>I’ve created a trivial model in python and saved it for mobile:</p>
<pre><code class="lang-auto">class SampleModel(torch.nn.Module):
    def forward(self, images):
        return torch.mean(images, dim=(1, 2, 3))

model = SampleModel()

scripted_model = torch.jit.script(model)
scripted_model._save_for_lite_interpreter(path)
</code></pre>
<p>in iOS I’m using trivial run logic:</p>
<pre><code class="lang-auto">torch::jit::mobile::Module model = ::torch::jit::_load_for_mobile(path);

c10::InferenceMode guard;
std::vector&lt;torch::jit::IValue&gt; inputs;

inputs.push_back(torch::ones({1, 1, 224, 224}, torch::kFloat32));

at::Tensor result = model.forward(inputs).toTensor();
</code></pre>
<p>This is as trivial an example as I’ve been able to build and I consistently get the EXC_BAD_ACCESS exception.  I tried saving a standard torchscript (non-mobile) and switching to <code>LibTorch</code> to see if it’s the mobile runtime, and I get an internal assert being thrown instead.</p>
<p>Anyone seen this or have thoughts on what I could be doing wrong?</p>
<p>Thanks in advance.</p>

username: beakin
cooked: <p>Resolved - moral of the story is make sure that there aren’t similarly named private variables in your c++ class.  Module was being assigned to the wrong one.  Please disregard…</p>

username: Linbin
cooked: <p>I didn’t see any obvious problem except “c10::InferenceMode guard” should be called before _load_for_mobile. Could you paste the full error message?</p>

username: beakin
cooked: <p>Couldn’t figure out how to close the post - in the full code the model was loaded into a variable that was in a different scope.  Dumb error - working now.  Thanks!</p>
Channel ordering [B, H, W, C] or [B, W,H,C]

username: ArshadIram
cooked: <p>Hi Community,</p>
<p>It is just an informative question that may be very basic to most of you. I was reading about channel ordering. Pytorch prefers the first channel order like this [B, C, H, W] and TensorFlow support channel last order [B, H, W, C]. So I am wondering to know does change the channel order affect performance?</p>
<p>What does PyTorch prefer first or last?<br>
How can we visualize these images?</p>
<p>Many thanks</p>

username: ptrblck
cooked: <p>PyTorch uses channels-first by default and allows you to transform the input as well as model parameters to channels-last as described <a href="https://pytorch.org/tutorials/intermediate/memory_format_tutorial.html">here</a>, which could be beneficial for mixed-precision training using TensorCores.</p>
<aside class="quote no-group" data-username="ArshadIram" data-post="1" data-topic="132787">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/arshadiram/48/32886_2.png" class="avatar"> ArshadIram:</div>
<blockquote>
<p>How can we visualize these images?</p>
</blockquote>
</aside>
<p>You can visualize the images using any library capable of it (e.g. <code>PIL</code> or <code>matplotib</code>).</p>

username: ArshadIram
cooked: <p>Thank you for always being available and quickly reply to every query.</p>