akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Get 2048 Length Feature Vectors of Inception Model TorchVision 0.7.0

username: casuru
cooked: <p>I am attempting to train an LSTM for predicting video frames. The frames will be represented as feature vectors pulled from a the outputs. I am trying to get the 2048 size features right before classification, but am struggling to do so. When I use IntermediateLayerGetter, I recieve this error, when trying to run the new model.</p>
<p>RuntimeError: Expected 4-dimensional input for 4-dimensional weight [192, 768, 1, 1], but got 2-dimensional input of size [1, 1000] instead</p>
<p>Any hints as to what I am doing wrong or how to get these features?</p>

username: ptrblck
cooked: <p>Could you post an executable code snippet, which shows your current approach?<br>
You should be able to get the intermediate activation using forward hooks as described <a href="https://discuss.pytorch.org/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/6">here</a>.</p>
[Torchscript/ONNX]: Howto to expose List Of List Tensors as input during onnx conversions

username: ThomasRetornaz
cooked: <p>Hi all and thanks if you read this</p>
<p>I have a special case of transformer where the input of the decoder is a list, and one of the element of the list is a list of 2D Tensors (size of list=batch size) and the ouput is a tuple(List Tensors)<br>
I’m able to convert my model in torchscript via script method and wrap the model<br>
Unfortinately i don’t know howto expose this kind of  input/ouput to torch.onnx.export</p>
<pre><code class="lang-auto">torch.onnx.export(
            decoder,
           (output_encoder,), &lt;----- Ouput Encoder is a list of 2d Tensors (with a dynamic dimentions)
            "decoder.onnx",
            input_names=["input"],
            output_names=["output"],
            dynamic_axes={"input": {0: {0: "sequence"}}, "output": {0: "sequence"}}, &lt;---- fail, howto define this use case ?
        )
</code></pre>
<p>Regards</p>
Is it necessary to use packed sequences for variable length RNN inputs, utilizing mini-batch training

username: Clint
cooked: <p>Hey all</p>
<p>I’m currently in the process of looking at padding time-series for an LSTM implementation of mine and I’m trying to wrap my head around the use of packed sequences. I’m wondering why I shouldn’t just pad my sequences with float(‘nan’), and then search for nan in the output and use that to filter my loss function? This may be specific for my case, but I only care about the final output at the end of my time-series, so, for example in the below code, could I not just use the known lengths of my sequences to pull out the values just before the 'nan’s and use that to calculate my loss?</p>
<pre><code class="lang-auto">In [66]: x = [[3,5,6,4,float('nan')],
    ...:     [3,5,4,float('nan'),float('nan')],
    ...:     [4,5,6,7,5]]

In [70]: xT = torch.tensor(x).view(3,5,1)

In [73]: lstm = nn.LSTM(1,3,1,batch_first=True)

In [75]: o,_ = lstm(xT)

In [77]: lin = nn.Linear(3,1)

In [78]: lin(o)
Out[78]: 
tensor([[[-0.2357],
         [-0.2539],
         [-0.2588],
         [-0.2561],
         [    nan]],

        [[-0.2357],
         [-0.2539],
         [-0.2551],
         [    nan],
         [    nan]],

        [[-0.2414],
         [-0.2560],
         [-0.2619],
         [-0.2622],
         [-0.2624]]])

</code></pre>
<p>My apologies if this is a stupid question, I’m still kind of new to pytorch and would like to understand this.</p>
<p>Thanks!</p>

username: nihal-rao
cooked: <p>Hey <a class="mention" href="/u/clint">@Clint</a> , did you find a solution ? I’m stuck on the same problem , I’m using a time series and cannot pad with 0.</p>
Activation Function

username: Mi_Rak
cooked: <p>Hello all, I am beginner in neural net, Just want to understand Activation Function.<br>
Is this applicable as activation function? By testing the two codes, the loss of the first one is not decreasing too much than the second one.</p>
<p><code>self.sigmoid(self.relu(x))</code></p>
<p>instead to</p>
<p><code>self.sigmoid(x)</code></p>

username: ayalaa2
cooked: <p>Essentially, any non-linearity we can introduce will work as an activation function. So there’s no reason why <code>sigmoid(relu(x))</code> can’t work. But some activation functions perform better than others.</p>

username: shartzog
cooked: <p>I’m pretty new to this stuff also, but if I understand correctly, the first implementation, i.e. sigmoid(relu(x)), would effectively constrain the output of the sigmoid between 0.5 and 1 rather than 0 and 1.  The relu function returns 0 for any value of x &lt;= 0 and the value of the sigmoid function as x–&gt;0 = 0.5 (it’s actually undefined AT x = 0, but every sigmoid implementation I’m aware of just papers that over so you don’t have to worry about it… :)).  For most models, I would think just sigmoid(x) would be more effective since sigmoid is often used as the final activation of a forward pass to return, for example, the “probability” that a sample possesses each label in an array of multiple labels, but as <a class="mention" href="/u/ayalaa2">@ayalaa2</a> said, it depends on your use case.</p>
<p><strong>EDIT:</strong> it’s actually NOT undefined at zero, but the rest still applies… <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: Mi_Rak
cooked: <p>Thank so much for your reply and for the explanation.</p>
CUDA memory leak

username: rajcscw
cooked: <p>Hi, I am trying to train several models in parallel using torch 's <code>pool.map()</code>. Since my setup has multiple GPUs, I pass a device also to my training task and the model is trained on that particular device.</p>
<p>The problem I face is <code>RuntimeError: CUDA error: out of memory </code><br>
after a while.</p>
<p>This happens after several models are trained and I can clearly see using <code>watch nvidia-smi</code> that the GPU memory accumulates over time.</p>
<p>I have posted a minimal example below which also leads to the stated issue.</p>
<p>Is there something needs to be done at the end of <code>run()</code> to clean up the memory?</p>
<p><strong>Pytorch version: torch 1.6.0</strong></p>
<pre><code class="lang-auto">import torch
from torch.multiprocessing import Pool
import random


class SimpleModule(torch.nn.Module):
    def __init__(self, input_: int, output: int):
        super().__init__()
        self.linear = torch.nn.Linear(input_, output)

    def forward(self, input_: torch.Tensor):
        return self.linear(input_)


def run(device: str):
    model = SimpleModule(10, 10).to(device)
    optimizer = torch.optim.SGD(lr=1e+2,
                                params=model.parameters())
    loss_criterion = torch.nn.MSELoss()
    for i in range(int(1e+4)):
        optimizer.zero_grad()
        inputs = torch.rand(5, 10).to(device)
        outputs = torch.rand(5, 10).to(device)
        preds = model.forward(inputs)
        loss = loss_criterion(preds, outputs)
        loss.backward()
        optimizer.step()


if __name__ == "__main__":
    torch.multiprocessing.set_start_method("spawn", force=True)
    n_tasks = 500
    available_device = ["cpu"] if not torch.cuda.is_available() else [f"cuda:{i}" for i in
                                                                      range(torch.cuda.device_count())]
    devices = random.choices(available_device, k=n_tasks)
    pool = Pool(processes=50)
    pool.map(run, devices)
</code></pre>
Install problem: pytorch 1.2.0 with Cuda 10.2

username: xen
cooked: <p>Hi,</p>
<p>I’m trying to run code from the following github repository: <a href="https://github.com/AnticipatedLearningMachine/Anticipated-Learning-Machine" rel="nofollow noopener">https://github.com/AnticipatedLearningMachine/Anticipated-Learning-Machine</a></p>
<p>In the requirements section at the bottom, it says that pytorch 1.2.0 and CUDA 10.2 are required. I have tried installing pytorch 1.2.0 (on Windows Anaconda) via pip, but I get a version that comes with Cuda 9.2. Can someone please help me install pytorch 1.2.0 with CUDA 10.2?</p>
<p>Thanks,<br>
Gavin.</p>

username: ptrblck
cooked: <p>I don’t think that the PyTorch <code>1.2.0</code> binaries and wheels were built with CUDA10.2, but with CUDA9.2 and CUDA10.0 as seen <a href="https://pytorch.org/get-started/previous-versions/#v120">here</a>.<br>
If you need this config for a particular reason, you would most likely have to build PyTorch from source.</p>

username: xen
cooked: <p>Great, thank you.</p>
<p>Gavin.</p>
MaskRCNN and Apex O2 issue

username: Walid_Ahmed_pyTorch
cooked: <p>I am trying to use O2 but roialign is giving me the following error<br>
"File “/opt/conda/lib/python3.7/site-packages/torchvision/ops/roi_align.py”, line 45, in roi_align<br>
sampling_ratio, aligned)<br>
RuntimeError: Expected tensor for argument <span class="hashtag">#1</span> ‘input’ to have the same type as tensor for argument <span class="hashtag">#2</span> ‘rois’; but type torch.cuda.HalfTensor does not equal torch.cuda.FloatTensor (while checking arguments for ROIAlign_forward_cuda)<br>
"<br>
Can you please guide me how to resolve this error?</p>
<p>Thanks<br>
Walid</p>

username: ptrblck
cooked: <p>Double post from <a href="https://discuss.pytorch.org/t/apex-memory-gains-in-detectron2/68122/8">here</a>.</p>
More GPU Memory Utilization on 2080Ti

username: sakura
cooked: <p>Hi, all. I have trained a detection model on 1080Ti, with PyTorch 1.0.1. The GPU memory in inference is about 500M. However, when I run the same model on 2080Ti, the GPU memory increases to about 850M. What is the reason and how can I reduce the extra ~350M memory in 2080Ti？</p>

username: ptrblck
cooked: <p>Could you create a single <code>torch.randn(1).cuda()</code> tensor and check the memory usage via <code>nvidia-smi</code>?<br>
If the ~350MB are visible, they are coming from the CUDA context and you cannot reduce it.<br>
The size of the context depends on the CUDA version, used device etc.</p>

username: sakura
cooked: <p>Hi, in my case, it costs 345MB memory on 1080Ti and 477MB on 2080Ti.</p>
<p>I also find another case: <a href="https://discuss.pytorch.org/t/2080ti-cost-more-memory-than-1080ti/45864" class="inline-onebox">2080ti cost more memory than 1080ti?</a>, which says that it costs 357MB memory on 1080Ti and 471MB on 2080Ti.</p>
<p>Moreover, when I just load model without input image, the memory costs on 1080 and 2080 already have ~300M gap.</p>

username: ptrblck
cooked: <p>The size of the CUDA context depends on the CUDA version as well as the device, so it’s unfortunately expected.</p>

username: sakura
cooked: <p>In other words, this is a normal phenomenon that we cannot avoid?<br>
And thank you very much for above detailed answers : )</p>

username: ptrblck
cooked: <p>Yes, you cannot avoid the overhead of the CUDA context unfortunately. <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>

username: sakura
cooked: <p>OK, understand.<br>
Thanks again for your help ~</p>

username: JeeLee
cooked: <p>How can I decrease the overhead of the CUDA context ?</p>

username: ptrblck
cooked: <p>One possibility would be to remove specific libraries, such as NCCL, cudnn etc., which should reduce the context, as less code has to be stored.<br>
Alternatively, you could try to remove PyTorch CUDA kernels, but this sounds pretty painful and you could be running in a lot of failures for undefined functions.</p>

username: JeeLee
cooked: <p>Thanks for your anwser.<br>
How can I remove specific libraries?Is that the right way to  build pytorch from source witout cudnn ?</p>

username: ptrblck
cooked: <p>Yes, you could specify the env vars (e.g. <code>USE_CUDNN=0</code>) and build PyTorch from source.</p>
Working with variable input sizes and increasing the batch_size, best approach?

username: tueboesen
cooked: <p>I’m working with a transformer network at the moment, so my input has various sizes. Currently I have just been training with a batch_size of 1, but I want to change this to something higher.</p>
<p>I see two possible ways of doing this and I was wondering what the pro’s/con’s of the two methods are which is the best? and whether there are any alternatives I have missed.</p>
<p>Method 1)<br>
I increase the batch_size in the dataloader, and will then have to apply some kind of padding transform to ensure that my data reaches a similar size in each batch.<br>
This will likely be relatively fast, since it will enable me to get multiple inputs through my network at once, but I guess I will have to include a padding mask in my transformer so it doesn’t learn anything from the padding, which I am not sure how well will work?</p>
<p>Method 2)<br>
I keep the batch_size at 1, but I accumulate the loss from several inputs before I compute the backprop and do an optimization step. In this way I won’t have to do any padding/masking. However I suspect it will be slower and there might be other pitfalls to this approach that I haven’t considered?</p>
<p>Does anyone have any experience with this? or good suggestions?</p>
Fourier transform

username: saandeep_aathreya
cooked: <p>I want to apply rfft to an image tensor of shape (60, 1, 256, 256). After the torch.rfft(img, signal_ndim=2) operation, the size of the output tensor is (60, 1, 256, 129, 2). Can someone please elaborate on this new output size?</p>
<p>Thanks in advance.</p>

username: Wesley_Neill
cooked: <p>Hi there! I’m going to make some assumptions about what your input dimensions represent and then tell you what the output represents based on those assumptions. I say this because I typically use the <code>librosa</code> library (highly recommended) for audio processing and have never used PyTorch’s <code>rfft</code>. However, they should be almost identical:</p>
<p><strong>Assumptions about your input:</strong></p>
<ul>
<li>60 - Batch Size</li>
<li>1 - Number of audio channels (mono in your case?)</li>
<li>256 - Amplitude</li>
<li>256 - Discrete-time samples</li>
</ul>
<p><strong>Output based on those assumptions:</strong></p>
<ul>
<li>60 - Batch size!</li>
<li>1 - The number of channels (mono)</li>
<li>256 - The amplitude scale of the DFT</li>
<li>129 - The Frequency scale of the DFT</li>
<li>2 - Probably corresponds to <code>signal_ndim = 2</code>
</li>
</ul>
<p><strong>A little explanation:</strong><br>
For real valued inputs, a DFT typically contains redundant information in the second half. Because of this <code>rfft</code> has a <code>onesided</code> flag that is set to <code>True</code> by default. What this means is that the returned DFT will be cut in half, having size <code>N/2 + 1</code>, which in your case appears to be <code>256/2 + 1 = 129</code>.</p>
<p><strong>For more information:</strong></p>
<p>I’d highly recommend researching DFT’s as a general topic before trying to use them in your code. Some topics to google are <strong>Nyquist-Shannon Sampling Theory</strong>, <strong>Sampling Rate vs Bandwidth (related to the first)</strong> and <strong>Discrete Fourier Transforms</strong>. <code>fft</code> is just a “fast” way to compute a DFT, so it’s helpful to first understand what a DFT is theoretically.</p>
<p>If you have any more specific questions, please let me know. I will try to answer them if I possibly can. I’m not an expert, but I’m an enthusiast.</p>

username: saandeep_aathreya
cooked: <p>Thank you for your detailed response <a class="mention" href="/u/wesley_neill">@Wesley_Neill</a>.  This gives me a good context to go deeper into the topic.</p>
Understanding CNN diagram

username: kolohe113
cooked: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/2/52203bdf1cffa781e8e4034bd6191a3f8caa1565.png" alt="image" data-base62-sha1="bIwde2TKBr2LRpmo7gP4ezk9sR7" width="527" height="235"><br>
Hi! I am trying to understand this diagram and I wonder if someone can help me. In the Conv layer, why did it have five layers?</p>

username: RaLo4
cooked: <p>Hard to say by just looking at the small image you copied.</p>
<p>I am guessing that the 4 in <strong>conv(4,4)</strong> refers to the <code>kernel_size</code> being 4x4. Which is a weird size since kernel size is usually always an odd number (check <a href="https://datascience.stackexchange.com/questions/23183/why-convolutions-always-use-odd-numbers-as-filter-size" rel="nofollow noopener">here</a> if you want to know why).<br>
The <strong>64 ch</strong> is probably the number of <code>out_channels</code>. This would mean tho that the drawn diagram should actually contain 64 of those grey boxes and not just 5. But I guess they just couldn’t be bothered to draw 64 of them.</p>
<p>Those are all just guesses/assumptions on what I can see here. If you could link the source, I could maybe give clearer answers.</p>

username: kolohe113
cooked: <p>Thank you for your help. Here is the source for the image:<br>
<a href="https://uav.berkeley.edu/2018/05/01/autoquad.html" class="onebox" target="_blank" rel="nofollow noopener">https://uav.berkeley.edu/2018/05/01/autoquad.html</a></p>

username: RaLo4
cooked: <p>Hmmm. The source is indeed not very specific about this.<br>
I also couldn’t find this particular model on their gihub page.</p>
<p>But I believe everything is as I said earlier:</p>
<aside class="quote no-group" data-username="RaLo4" data-post="2" data-topic="94955" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/65b543/48.png" class="avatar"> RaLo4:</div>
<blockquote>
<p>Hard to say by just looking at the small image you copied.</p>
<p>I am guessing that the 4 in <strong>conv(4,4)</strong> refers to the <code>kernel_size</code> being 4x4. Which is a weird size since kernel size is usually always an odd number (check <a href="https://datascience.stackexchange.com/questions/23183/why-convolutions-always-use-odd-numbers-as-filter-size" rel="noopener nofollow ugc">here</a> if you want to know why).<br>
The <strong>64 ch</strong> is probably the number of <code>out_channels</code> . This would mean tho that the drawn diagram should actually contain 64 of those grey boxes and not just 5. But I guess they just couldn’t be bothered to draw 64 of them.</p>
<p>Those are all just guesses/assumptions on what I can see here. If you could link the source, I could maybe give clearer answers.</p>
</blockquote>
</aside>
Remove padding dimension from output layer

username: ebelle
cooked: <p>I built a POS tagger and it’s having trouble learning (which is anticipated given the small dataset and large number of tags). To help, I’d like to remove the padding dimension from the output layer, so it stops predicting padding as one of the tags. Right now, pad_idx=0 and I’m wondering if there’s an easy (and reusable) way to always make the pad_idx the last element in the vocabulary so that I can just make output_dim=len(TRG.vocab)-1? Or is there a more elegant approach to this?</p>

username: ebelle
cooked: <p>I ended up not having a padding token in the vocabulary and then I padded the batch with whatever index was len(TRG.vocab)+1. Whenever I needed the pad_idx (for ignore_idx in the criterion for example) I just used len(TRG.vocab)+1. Note that I built my own Dataset and Loader so this may not work if you’re not padding your batches yourself.</p>
LBFGS converges but stops early

username: Zador
cooked: <p>Hi,</p>
<p>I am new to working with the LBFGS optimizer and and have found to to get incredibly impressive results. In cases I have applied it to it, it has far outperformed other optimizers such as Adam. When applying it to a new and far more complex case to the previous ones it also converges impressively, however, when getting close to the solution that I am looking for, the training process stops.</p>
<p>I have done some research and some experimentation with the LBFGS method and have concluded that this premature termination can be altered by <strong>tolerance_grad</strong> and <strong>tolerance_change</strong>. I have set these to 1e-200 and 1e-400 respectively, and it is true that the process lasts longer, but nevertheless it still terminates prematurely. Is there any way I can turn this automatic termination off, and replace it with a manually assigned termination condition? Perhaps setting the tolerance grad and change equal to 0 and adding the termination requirement into the optimizers closure function instead?</p>
<p>Thanks.</p>

username: Abueidda
cooked: <p>Hi,</p>
<p>I am encountering a similar scenario. Did you find any thing interesting to resolve such an issue? Thanks for the post.</p>

username: Zador
cooked: <p>Hi,</p>
<p>sorry for my late response. I did not have the time to look into the algorithm properly to give a reason for the things I found but I found that by setting the tolerances to zero the algorithm stops when it no longer learns and be setting the tolerances to a negative number, the algorithm does not terminate although it no longer learns. The latter was clear from analyzing the running loss while the former I deduced from the latter.</p>
<p>The algorithm implements line search so I suppose this makes sense.</p>
Accumulating a tensor in a memory efficient way

username: Haxxardoux
cooked: <p>I am implementing an attention-like mechanism autoregressively, which means I will need to accumulate a tensor containing all previous GRU hidden states and do a matrix multiplication at each timestep. To accumulate this tensor, I could use .cat(), but as memory is likely to be the bottleneck during training, this seems a bit sketch.</p>
<p>My current solution (does not work) was to pre-allocate a tensor with zeros, then just assign hidden states to it at each timestep, while simultaneously using it in a matrix multiplication operation. However, this returns an error that one of the variables needed for gradient computation has been modified by an inplace operation, as expected.</p>
<p>My question is, how would one accumulate a tensor in this way in a memory-efficient way? In the paper that introduced the technique I am using (SHA-RNN), this was praised as being computationally efficient due to the way you can accumulate this tensor and not repeat computations, but I am not seeing how this is possible at the moment.</p>

username: Haxxardoux
cooked: <p>Update: The SHA-RNN codebase, which claimed to be very memory and computationally efficient, just uses torch.cat. I have been decieved.</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/Smerity/sha-rnn/blob/master/model.py#L209" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/Smerity/sha-rnn/blob/master/model.py#L209" target="_blank" rel="nofollow noopener">Smerity/sha-rnn/blob/master/model.py#L209</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="199" style="counter-reset: li-counter 198 ;">
<li>    #x = x + z.sum(dim=-2)</li>
<li>
</li><li>    h = h + x if self.residual else x.float()</li>
<li>
</li><li>focus, new_mem = None, []</li>
<li>if self.attn is not None:</li>
<li>    mh = self.lnmem(h)</li>
<li>    h = self.lnmid(h)</li>
<li>
</li><li>    if mem is not None:</li>
<li class="selected">        bigh = torch.cat([mem, mh], dim=0)</li>
<li>    else:</li>
<li>        bigh = mh</li>
<li>    new_mem = bigh[-len(pe):]</li>
<li>
</li><li>    q, k = h, bigh</li>
<li>
</li><li>    x, focus = checkpoint(self.attn, q, k, bigh, attn_mask)</li>
<li>    #x, focus = tcheckpoint(self.attn, q, k, bigh, attn_mask)</li>
<li>    x = self.drop(x)</li>
<li>    h = x + h</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Automatic Bounding Box on simple Background

username: TheDoctor
cooked: <p>Hi,</p>
<p>i have a very simple background and an object laying on it. Im trying to find a pretrained model, which can  propose a bounding box without me having to specify it beforehand. So by just using jpg files.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/b/dbfb8da6b14bde7e65ebefb1380ed795590e1cc4.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/dbfb8da6b14bde7e65ebefb1380ed795590e1cc4" title="IMG_20200523_023411_BURST57"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/b/dbfb8da6b14bde7e65ebefb1380ed795590e1cc4_2_375x500.jpeg" alt="IMG_20200523_023411_BURST57" data-base62-sha1="vo3w1AcjlShE91T88Dqw2kx6Khe" width="375" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/b/dbfb8da6b14bde7e65ebefb1380ed795590e1cc4_2_375x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/d/b/dbfb8da6b14bde7e65ebefb1380ed795590e1cc4_2_562x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/b/dbfb8da6b14bde7e65ebefb1380ed795590e1cc4_2_750x1000.jpeg 2x" data-dominant-color="B5A394"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">IMG_20200523_023411_BURST57</span><span class="informations">960×1280 63 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> .</p>
<p>Does there exist a model for that which can do it?</p>
Training loss and validation loss graph

username: Gutabaga
cooked: <p>Hello, am trying to draw graph of training loss and validation loss using matplotlip.pyplot but i usually get black graph.</p>
<p>my code is like this<br>
plt.plot(train_loss, label=‘Training loss’)<br>
plt.plot(valid_loss, label=‘Validation loss’)<br>
plt.legend(frameon=False)</p>
<p>and the code which produce those loss value is<br>
n_epochs = 30<br>
valid_loss_min = np.Inf</p>
<p>for epochs in range(1, n_epochs+1):</p>
<pre><code>train_loss = 0.0
valid_loss = 0.0

model.train()
for images, labels in train_loader:
    optimizer.zero_grad()
    output = model(images)
    loss = criterion(output,labels)
    loss.backward()
    optimizer.step()
    
    
    train_loss += loss.item()
    
model.eval()
for images, labels in valid_loader:
    output = model(images)
    loss = criterion(output,labels)
    
    valid_loss += loss.item()
train_loss = train_loss/len(train_loader.dataset)
valid_loss = valid_loss/len(valid_loader.dataset)

print('Epoch:  {}. Training Loss: {:.6f}. Validation_loss: {:.6f}'.format(epochs,train_loss,valid_loss))

if valid_loss &lt;= valid_loss_min:
    print( 'Varidation Loss is decrease: ( {:.6f} --&gt;{:.6f}). save the model...'.format(valid_loss_min, valid_loss))
    torch.save(model.state_dict(), 'mlp2_model.pt')
    valid_loss_min = valid_loss</code></pre>

username: Rahul_pillai
cooked: <p>it should be list of values. Try creating training_loss list and validation loss list out of the for loop<br>
and append the training and validation loss to it.</p>

username: pourya_farzi
cooked: <aside class="quote no-group" data-username="Gutabaga" data-post="1" data-topic="37194">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/g/47e85d/48.png" class="avatar"> Gutabaga:</div>
<blockquote>
<p><code>    train_loss += loss.item()</code></p>
</blockquote>
</aside>
<p>Your  mistake lies in here<br>
train_loss += loss.item()</p>
<p>correct it with this one for both loops<br>
train_loss += loss.item().len(images)</p>
Sparse Cross Entropy Loss for calculating Loss in NLP problems

username: tiwari-ai-harsh
cooked: <p>My Input tensor Looks like torch.Size([8, 23])<br>
8 - batch size, with 23 words in each of them<br>
My output tensor Looks like torch.Size([8, 23, 103])<br>
8- batch size, with 23 words predictions with 103 vocab size.</p>
<p>I want to calculate sparse cross Entropy Loss for this task, but I can’t since PyTorch only calculates the loss single element. How can I code it to work? Thanks for your help.</p>
Tracking tensor transformations through a module

username: Valiox
cooked: <p>I’m working with a fairly large model, and I frequently run into issues where I unwillingly change the output of the model when working in a dev branch.</p>
<p>I’d like to avoid that by building a unit test that will track the values of each intermediate tensor built in the forward pass, and compare those values with that of the current master branch. With a deterministic network (or a fixed seed) and a fixed input, that should help in preventing silent bugs.</p>
<p>How can I go about doing this? Is it possible? Does it involve tracing?</p>
Can't access intermediate pre-activation values in a network

username: amswag
cooked: <p>Hi all,<br>
I am trying to get intermediate layer values for a pretrained network, say VGG19. The main issue is to access pre-activation values i.e in the case of ReLU, I need both the positive and negative values just before the activation layer.</p>
<ol>
<li>I am able to attach forward hooks and then do a single forward pass(after attaching all forward hooks to reqd layers) but that gives me only positive values ( I suspect this is because of the in-place ReLU operation but not sure) except for the last layer where no ReLU follows it.</li>
<li>In another implementation, if I try to do a forward pass after every register_forward_hook, I am able to get the negative values as well, but I am not able to get gradients with respect to the image( I need those as well). I suspect that it might be due to multiple forward passes being done, but would like some clarification if wrong.</li>
</ol>
<p>Any help would be appreciated. Do let me know if some code is required due to any ambiguity in the question asked. Thanks.</p>

username: ptrblck
cooked: <ol>
<li>
<p>You are most likely right and the <code>inplace</code> <code>ReLU</code> would manipulate the output of the incoming activation. You could copy-paste the VGG implementation and change <a href="https://github.com/pytorch/vision/blob/195c41ee7ae3fff8f64a4fda67632259dd4f7454/torchvision/models/vgg.py#L32-L35">these</a> and <a href="https://github.com/pytorch/vision/blob/195c41ee7ae3fff8f64a4fda67632259dd4f7454/torchvision/models/vgg.py#L71-L74">these lines of code</a> to <code>nn.ReLU</code> or alternatively replace them manually in the model.</p>
</li>
<li>
<p>I’m not sure I understand the question correctly. The gradients would be calculated for the parameters by default (not the activations). Which gradients would you like to calculate?</p>
</li>
</ol>
Contractive Loss rising in CAE - Contractive Autoencoder

username: Zach_scary
cooked: <p>Hello,<br>
I am builing a Contractive Autoencoder. My MSELoss is decreasing, but unfortunately my Contraction Loss is rising, wheres it should be different.</p>
<p>Did anyone encounter similar issues?</p>
<p>Here is my Loss Function for the Contractive Loss: I use LeakyReLU as activation and MSE.</p>
<p>def loss_function(W, x, predicted, h, lam):<br>
mse = loss_crit(predicted, x)</p>
<pre><code>dh = torch.where(h &gt;= 0 , torch.ones(1), torch.ones(1)*1e-2) 

w_sum = torch.sum(W**2, dim=1)

w_sum = w_sum.unsqueeze(1)

contractive_loss = torch.sum(torch.mm(dh**2,w_sum), 0)

return mse + contractive_loss * lam, mse, contractive_loss</code></pre>
Is any case to prefer detach().cpu() to cpu().detach(),or vice versa?

username: AlexLuya
cooked: <p>Hello,they give same result,any particular case to prefer detach().cpu() to cpu().detach(),or vice versa?</p>

username: ptrblck
cooked: <p>I would prefer  to use <code>detach().cpu()</code>, as this would detach the tensor already on the GPU and the <code>cpu()</code> operation won’t be tracked by Autograd. That being said, it seems to be a minor change and you shouldn’t see any differences in using any of these two approaches. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: AlexLuya
cooked: <p>Thanks,is there any case to prefer with cpu() than detach()?</p>

username: ptrblck
cooked: <p>I don’t understand the question. The <code>cpu()</code> operation transfers the tensor to the CPU (if not already there), while <code>detach()</code> cuts the computation graph. Could you explain the question a bit more?</p>
Implementing multi softmax layer for byte-level autoencoder

username: szyolewn
cooked: <p>Hello,</p>
<p>I want to implement a denoising autoencoder for typos correction tasks, which works on byte-level (utf-8 encoded strings), not on charter level. As a baseline, I want to create a simple dense autoencoder (see the picture below) with multi-softmax layer at the output. Each softmax represents a one byte of output - one-hot encoded. Can I achieve this behavior using <code>CrossEntropyLoss()</code> or do I need to do it diffirently?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/0/7033fdca23438c292b0e22b0cb5250fd4ccf7190.png" data-download-href="https://discuss.pytorch.org/uploads/default/7033fdca23438c292b0e22b0cb5250fd4ccf7190" title="multi-softmax"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/0/7033fdca23438c292b0e22b0cb5250fd4ccf7190_2_358x500.png" alt="multi-softmax" data-base62-sha1="g0ARNlP6DMsTY5zR12U83frYnmM" width="358" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/0/7033fdca23438c292b0e22b0cb5250fd4ccf7190_2_358x500.png, https://discuss.pytorch.org/uploads/default/original/3X/7/0/7033fdca23438c292b0e22b0cb5250fd4ccf7190.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/0/7033fdca23438c292b0e22b0cb5250fd4ccf7190.png 2x" data-dominant-color="BEBEBE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">multi-softmax</span><span class="informations">382×533 83.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: tom
cooked: <p>CrossEntropyLoss is strictly a combination of LogSoftmax and NLLLoss, so it’ll neither help nor hinder you. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
If the softmaxes all are over the same number of elements, you could use the k-dimensional variant, though.</p>

username: szyolewn
cooked: <p>Thank you. I was reading about k-dimentional variant but I’m not quite sure how it works. Can you give me some example? In the output I have the vector of size (batch, n_bytes, max_bytes_in_string), where n_bytes=256 and max_bytes_in_string=28. I want each byte of output being processed by separate softmax. How can I achieve that?</p>
Implementation of Natasha2 alg by Allen-Zhu

username: seeker_after_truth
cooked: <p>Hello! I wanted to ask if anyone knows of an implementation of the Natasha2 algorithm (introduced in this paper by Zeyuan Allen-Zhu: <a href="https://arxiv.org/abs/1708.08694" rel="nofollow noopener">https://arxiv.org/abs/1708.08694</a>)<br>
Natasha2 uses the Oja’s algorithm to avoid saddle points and converge to a local minimum for non-convex functions.</p>

username: Abhilash_Nandy
cooked: <aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">
      <a href="https://github.com/ruoshiliu/Natasha2" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars0.githubusercontent.com/u/25470644?s=400&amp;v=4" class="thumbnail onebox-avatar" width="400" height="400">

<h3><a href="https://github.com/ruoshiliu/Natasha2" target="_blank" rel="nofollow noopener">ruoshiliu/Natasha2</a></h3>

<p>Implementation of Natasha1.5 and Natasha2 algorithm in PyTorch - ruoshiliu/Natasha2</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Can't convert tensor with dtype float64 to torch.tensor.double

username: tonsoffun
cooked: <p>Hi,<br>
i want to convert a numpy array into a tensor with dtype double because it gives me the following error when calling:</p>
<pre><code class="lang-auto">wav_stacked = torch.tensor(wav_stacked)
for wave in wav_stacked:
    torchaudio.transforms.MelSpectrogram(downsample_rate)(wave.double())
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/rudolpher68005/.local/lib/python3.8/site-packages/torchaudio/transforms.py", line 427, in forward
    mel_specgram = self.mel_scale(specgram)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/rudolpher68005/.local/lib/python3.8/site-packages/torchaudio/transforms.py", line 260, in forward
    mel_specgram = torch.matmul(specgram.transpose(1, 2), self.fb).transpose(1, 2)
RuntimeError: Expected object of scalar type Double but got scalar type Float for argument #3 'mat2' in call to _th_addmm_out
</code></pre>
<p>How to convert the tensor so I can use it in the MelSpectogram function?</p>

username: tonsoffun
cooked: <p>The error was a little bit missleading. He expected a float not a double</p>
Data Distributed Parallel runs faster when all the cards are occupied

username: Haizhou_Shi
cooked: <p>Hi I’m rather new to the DDP, and I found a rather bizarre behavior of my training with DDP.</p>
<p>Say I have 4 GPUs in total. If I run my code on the GPU:0 and GPU:1 and leave the remaining two unoccupied. Then during training the percentage of both GPU would be at maximum 50%. Now the GPU occupation is:</p>
<p>gpu0: process1(50% or lower)<br>
gpu1: process1(50% or lower)<br>
gpu2: empty<br>
gpu3: empty</p>
<p>But when I run another training code on the remaining two GPUs, all 4 cards would hit 100% usage and both of the processes run faster than the previous situation. Now the gpu occupation is:</p>
<p>gpu0: process1 (99%)<br>
gpu1: process1 (99%)<br>
gpu2: process2 (99%)<br>
gpu3: process2 (99%)</p>
<p>I experience this on multiple servers and it’s really confusing me. Can anyone help with this?</p>

username: mrshenli
cooked: <p>How about the per iteration latency? If you feed the same batch size to each DDP instance (i.e., different global <code>batch_size</code>), is using 4 GPU still faster? Please use the <a href="https://pytorch.org/docs/stable/cuda.html#torch.cuda.Event.elapsed_time" rel="nofollow noopener"><code>elapsed_time</code></a> API to measure that.</p>
<p>As DDP uses <code>all_reduce</code> to communicate gradients, GPU utilization cannot faithfully represent how busy a GPU is. CUDA would report 100% GPU utilization even if one GPU is block waiting for another peer to join and doing nothing.</p>

username: Haizhou_Shi
cooked: <p>Thanks for your reply!</p>
<p>I didn’t use <code>elasped_time</code> to measure how long it takes for one iteration but I did use <code>time.time()</code> to calculate the iteration latency. As long as there is one GPU that remains unused, the process runs slower (sometimes 50% slower and sometimes 100% slower).</p>
<p>The weird thing is that it seems my DDP program is influenced by other programs (they are not necessarily DDPs) running on other GPUs, in a counterintuitive way: normally we expect that programs are competing for the computational resources, but here more programs bring better performance…</p>
<p>(This thing didn’t happen to my colleague’s non-DDP program so I guess it must have something to do with the DDP).</p>
Where is grad_fn set in pytorch source?(version 1.6)

username: AlexLuya
cooked: <p>Hello,we have a framework which will be responsible for doing configuration,start/stop training,etc,and it can be used to run another model training without any problem,so this framework seems work fine,but when it used to train my current model,the first epoch runs fine,so this model seems work fine also,but when the second epoch just begin,an exception got thrown out:</p>
<pre><code class="lang-auto">  File "/usr/local/lib/python3.8/dist-packages/torch/tensor.py", line 185, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/usr/local/lib/python3.8/dist-packages/torch/autograd/__init__.py", line 125, in backward
    Variable._execution_engine.run_backward(
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
</code></pre>
<p>by debugging,I found that  the output tenor of network has grad_fn = None,and this is reproduciable: always comes in FIRST backwarding of SECOND epoch.</p>
<p>Since this framework can be used to train other modes without any problem,and for current model,the first epoch runs fine also,so I want to dig deep to find out why grad_fn doesn’t got set,and after searching the source code of pytorch,I found it seems like being set  in nn.modules.module.py line 736</p>
<pre><code class="lang-auto">            grad_fn = var.grad_fn
            if grad_fn is not None:
</code></pre>
<p>but debugging doesn’t confirm my searching,so would you please tell where is grad_fn got set,thanks?</p>

username: ptrblck
cooked: <p>I would recommend to check the Python code, especially the forward pass, which would create the <code>grad_fn</code>s for all tensors, where the gradient calculation is not disabled.<br>
Based on your description, did you use <code>torch.set_grad_enabled</code> in any part of the code or a <code>with torch.no_grad()</code> wrapper accidentally for some epochs?</p>

username: AlexLuya
cooked: <p>Thanks,in may case,‘self.requires_grad_(False)’  caused this problem</p>
Cuda initialisation stuck in a100 machine

username: gouthamvgk
cooked: <p>I am using a A100 server from GCP with the latest NGC container from nvidia. However for the support of DCNV2 i have to downgrade my pytorch version to 1.4.0. Whenever i initialise a tensor in gpu like torch.randn(3).cuda() the interpreter gets stuck and never finishes that command. Any help??</p>

username: mrshenli
cooked: <p>hey <a class="mention" href="/u/ptrblck">@ptrblck</a>, do you know if anyone can answer questions regarding using PyTorch cuda features on GCP/NGC?</p>
<p>cc <a class="mention" href="/u/ngimel">@ngimel</a></p>

username: gouthamvgk
cooked: <p>Solved!. After close to 10 mins the tensor gets initialised in gpu and from thereon no problem</p>

username: ptrblck
cooked: <p>The long startup time is most likely create due to a JIT compilation of the CUDA code, if your installed PyTorch version wasn’t built for compute capability 8.0 (A100).<br>
This would be the case, if you’ve installed the <code>1.4</code> binary instead of building it from source.<br>
We are working towards building the PyTorch nightly binaries with the latest library stack and cc8.0.<br>
For now, you could either build from source or let the JIT compiler run in the first CUDA call.</p>

username: ngimel
cooked: <p>What is the issue with dcnv2 support in the latest ngc container?</p>

username: Shenggan
cooked: <p>Thanks for the reply. But I wonder that is it possible to build from source for old version PyTorch (for example version 1.4) with cuda11? Or is there any plan to support old version PyTorch for A100?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Shenggan" data-post="6" data-topic="90462">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shenggan/48/28294_2.png" class="avatar"> Shenggan:</div>
<blockquote>
<p>Or is there any plan to support old version PyTorch for A100?</p>
</blockquote>
</aside>
<p>There is no plan on changing older PyTorch versions to enable CUDA11 and thus new GPU architectures, so you would have to use the latest PyTorch version.</p>
<aside class="quote no-group" data-username="Shenggan" data-post="6" data-topic="90462">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shenggan/48/28294_2.png" class="avatar"> Shenggan:</div>
<blockquote>
<p>is it possible to build from source for old version PyTorch (for example version 1.4) with cuda11?</p>
</blockquote>
</aside>
<p>You could try to cherry-pick all commits mentioning CUDA11 in an older version and try to build it.<br>
However, while it might work, what’s your use case that you need to use an old PyTorch version?</p>

username: Shenggan
cooked: <p>Thanks for the reply. I think porting to the latest version PyTorch is the best choice.</p>
Allow subset of layer weights to be updated

username: RTC
cooked: <p>I have a network and after a few conv layers, it has 3 heads: <code>self.fc_pi</code>, <code>self.fc_q</code> and <code>self.fc_beta</code>. I have frozen all of the conv layer weights and am allowing updates only to <code>self.fc_q</code> and <code>self.fc_beta</code> by adding only their weights to the optimizer as:</p>
<pre><code class="lang-auto">self.optimizer = config.optimizer_fn(self.network.fc_q.parameters())
self.optimizer.add_param_group({'params': self.network.fc_beta.parameters()})
</code></pre>
<p>The <code>self.fc_pi</code> is a linear layer that connects 200 neurons to 16 neurons so when I do <code>print(list(self.network.fc_pi.parameters())[0].shape)</code>, I get [16,200] and when I do <code>print(list(self.network.fc_pi.parameters())[1].shape)</code>, I get [16], both of which respectively represent the weights and biases of this linear layer.</p>
<p>I would like to add a subset of these weights and biases to the optimizer so that only the weights connected to the first 4 (out of the 16 neurons), and their biases, are updated. I have tried this:</p>
<pre><code class="lang-auto">self.optimizer.add_param_group({'params': list(self.network.fc_pi.parameters())[0][0:4]})
self.optimizer.add_param_group({'params': list(self.network.fc_pi.parameters())[1][0:4]})
</code></pre>
<p>but this raises the error:</p>
<pre><code class="lang-auto">ValueError: can't optimize a non-leaf Tensor
</code></pre>
<p>How can I correctly add a subset of the layer weights and biases to the optimizer so that only a subset of weights is updated and not all the weights in the layer.</p>

username: mariosasko
cooked: <p>After the backward call, and before the optimizier update, you can access the gradient tensor (<code>param.grad</code>) and assign zero to the parts that you want to freeze/not update. This way these weights are left unchanged after the <code>optim.step</code> call.</p>
Tool to manipulate checkpoints

username: sharvil
cooked: <p>Hey folks, I just built tool to quickly edit and inspect checkpoints called <a href="https://github.com/lmnt-com/graftr" rel="nofollow noopener">graftr</a>.</p>
<p>It’s super useful if you’ve refactored some code and want to update your old checkpoints to match, or if you just want to peek inside a pre-trained model. Feedback is welcome and appreciated!</p>
Retrieve the device type and device ID from c++ programmatically in a JIT GraphPass

username: vdantu
cooked: <p>Is there a way to retrieve the device type and device ID of a model, from inside a custom JIT graph pass? In other words, if a customer does a <code>model.to("cuda:0")</code>, is there any way to retrieve “cuda, 0” from inside a graph pass? Please let me know.</p>

username: tom
cooked: <p>This depends on the state of the graph.<br>
For example, in a freshly traced graph, you have the values (mostly) with as complete tensor type and that includes the device. Similarly dimensioned tensors (with scalar type, dimension (1d 2d 3d etc) and device, requires grad) have that information, but more incomplete values don’t.</p>
<p>Best regards</p>
<p>Thomas</p>

username: vdantu
cooked: <p>Can be done in a custom graph pass during inference. Graph pass only has reference to <code>std::shared_ptr&lt;Graph&gt;</code> . But assuming that this is during inference, is it possible to retrieve the device-type and device-id (in case of cuda)? This could be scripted model or traced model.</p>