akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
RuntimeError: scatter_add_cuda_kernel does not have a deterministic implementation

username: aaditya
cooked: <p>I am running a RNN (GRU) network on Cuda on colab, getting this error.</p>
<p><code>RuntimeError: scatter_add_cuda_kernel does not have a deterministic implementation, but you set 'torch.use_deterministic_algorithms(True)'. You can turn off determinism just for this operation if that's acceptable for your application. You can also file an issue at https://github.com/pytorch/pytorch/issues to help us prioritize adding deterministic support for this operation.</code></p>
<p>I also tried the solution mentioned on official Pytorch doc</p>
<pre><code class="lang-auto">On CUDA 10.1, set environment variable `CUDA_LAUNCH_BLOCKING=1`. This may affect performance.

On CUDA 10.2 or later, set environment variable (note the leading colon symbol) `CUBLAS_WORKSPACE_CONFIG=:16:8` or `CUBLAS_WORKSPACE_CONFIG=:4096:2`.
</code></pre>
<p>But getting the same error.</p>
<p>There is no error while running the same code on the CPU.</p>
<p>What is a possible solution to this error?</p>
<p>Cuda version</p>
<pre><code class="lang-auto">nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2020 NVIDIA Corporation
Built on Mon_Oct_12_20:09:46_PDT_2020
Cuda compilation tools, release 11.1, V11.1.105
Build cuda_11.1
</code></pre>

username: ptrblck
cooked: <p>The feature request is tracked <a href="https://github.com/pytorch/pytorch/issues/50469">here</a>. As a workaround you could use the CPU implementation or try to replace <code>scatter_add</code> with another indexing operation, if possible. In any case, feel free to update the linked issue with your use case.</p>
Unable to use CUDA after it fails due to an OOM error

username: Cow_woC
cooked: <p>Hi,</p>
<p>I am attempting to train a model on the GPU and if the operation fails due to an out of memory error, I want to reduce the batch size and try again. I posted a testcase at <a href="https://github.com/pytorch/pytorch/issues/65301#issuecomment-922601567" class="inline-onebox" rel="noopener nofollow ugc">Anomaly detection: Error detected in CudnnRnnBackward0 · Issue #65301 · pytorch/pytorch · GitHub</a> showing that retrying the operation always fails with <code>CUDA error: an illegal memory access was encountered</code>.</p>
<p>I suspect there is a bug in the pytorch C++ implementation but I can’t be certain without further investigation by a pytorch committer.</p>
<p>Is anyone else able to reproduce this error using the testcase?</p>
<p>Am I supposed to somehow reset the CUDA state after an out of memory error?</p>
<p>Is this code doing anything wrong?</p>
<p>Thank you,<br>
Gili</p>

username: arya47
cooked: <p>Not sure about illegal access, but check how much of your dataset your are loading and how much memory your model requires.</p>
Classification loss for "any class but class X"?

username: ZimoNitrome
cooked: <p>I am trying to train a classifier on a set of classes.</p>
<p>I then want to use the classifier to generate a loss such that given a sample, the loss will only be high if the classifier returns a specific unwanted class. It is preferable that the classifier predicts another class with high “probability”/“confidence”, but it does not matter what the class is, as long as it is not the specified unwanted class.</p>
<p>In my initial attempt I used CELoss to train the classifier and attempted to use negative of that loss <code>-CELoss</code> to achieve my goal. But this seems to give the lowest possible loss when all classification scores are at 0.5.</p>
<p>Any ideas on how this can be achieved?</p>

username: KFrank
cooked: <p>Hi Zimo!</p>
<p>Why not train your classifier to predict “class X” and then just<br>
post-process your prediction to predict some other class?</p>
<aside class="quote no-group" data-username="ZimoNitrome" data-post="1" data-topic="132346" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zimonitrome/48/27591_2.png" class="avatar"> ZimoNitrome:</div>
<blockquote>
<p>the loss will only be high if the classifier returns a specific unwanted class.</p>
</blockquote>
</aside>
<p>Is <em>all</em> you care about that you get a high loss only for “class X?”</p>
<aside class="quote no-group">
<blockquote>
<p>It is preferable that the classifier predicts another class with high “probability”/“confidence”</p>
</blockquote>
</aside>
<p>Or do you also want just one non-class-X class to be predicted (rather<br>
than a muddle of multiple non-class-X classes)?</p>
<p>I don’t really understand your use case.  What do your training data<br>
look like, and, in particular, what are the “ground-truth” labels for your<br>
training data?  Is each training sample labelled with its corresponding<br>
“class-X?”  Or is it labelled with the actual class of the sample,<br>
together with the particular class-X that is evil for that sample?</p>
<p>Once you have your trained classifier together with any additional<br>
post-processing, what do you want the output of your overall “system”<br>
to be?  At “inference” time, I give you a previously-unseen sample.<br>
What output do you want your system to produce?</p>
<p>Best.</p>
<p>K. Frank</p>
Patch a full dataset folder images

username: Najeh_Nafti
cooked: <p>How can I create patches of size (64,64) for images with 3 channels and save them from a dataset of size (256,256).</p>

username: ptrblck
cooked: <p>You could use <a href="https://pytorch.org/docs/stable/generated/torch.Tensor.unfold.html"><code>tensor.unfold</code></a> to create the patches and save them via <code>torch.save</code>.</p>

username: Najeh_Nafti
cooked: <p>How <code>tensor.unfold</code> perform patching, is it randomaly?<br>
And then how can I reconstruct the original images?</p>

username: ptrblck
cooked: <p>No, it’s not randomly and the example in the docs would show how the kernel size and stride is applied.<br>
To reconstruct the input, you could take a look at the <a href="https://pytorch.org/docs/stable/generated/torch.nn.Unfold.html?highlight=unfold#torch.nn.Unfold"><code>nn.Unfold</code></a> and <a href="https://pytorch.org/docs/stable/generated/torch.nn.Fold.html?highlight=fold#torch.nn.Fold"><code>nn.Fold</code></a> docs.</p>
Gradient w.r.t target values

username: Tejas_Pagare
cooked: <p>Hi, I wanted to ask if PyTorch considers gradient w.r.t target in it’s computation. Consider the following example:<br>
Here the target as well as output both comes from the same network</p>
<pre><code class="lang-auto">net = torch.nn.Linear(2,2)
input = torch.tensor([1.,0.])
out = net(input)
target = net(torch.tensor([2.,2.]))
loss = nn.functional.mse_loss(out,target)
optimizer.zero_grad()
loss.backward()
optimizer.step()
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/4/a4624b536d15cf782858eaede81932bb6c1d4ad1.png" alt="image" data-base62-sha1="nscTM2KsMVyNZVKYFdDd2AvYNz3" width="414" height="84"></p>
<p>Does the above code translates to the gradient update of 1 or 2?<br>
If it translates to 2 then how should I implement so that the gradient update is as given in 1?<br>
Basically I want to update network parameters considering the gradient wrt to both out as well as target.</p>
<p>Thanks!<br>
Any help would be really appreciated.</p>

username: Unity05
cooked: <p>Hi,<br>
yes, PyTorch considers gradients w.r.t. the output as well as the target in your case.</p>

username: Tejas_Pagare
cooked: <p>Any quick way to check this?</p>

username: Unity05
cooked: <p>Using backward hooks. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=10" title=":wink:" class="emoji" alt=":wink:"></p>

username: Tejas_Pagare
cooked: <p>Hey hi, thanks a lot! Used hooks and did some maths and yeah you’re correct. Idk why at first instance I thought that <code>mse_loss</code> treats/expects targets as constant.</p>

username: Unity05
cooked: <p>Hi,<br>
glad to hear you’ve been able to confirm my answer! Don’t feel bad, you’ve learned something! <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=10" title=":wink:" class="emoji" alt=":wink:"> <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
How to sample from a vector

username: Xia_Zhengmi
cooked: <p>Hi everyone,</p>
<p>I have two different vectors as the inputs of my model. The first is an EEG signal and the second one is a vector of desired locations just on which I want to do convolution. My location vector consists of binary values (1 or 0).</p>
<p>Besides, is there any possibility to use the location vector in the form of a=[1 3 2 1 2 2 2 3 1 1] and then chose different branches to do the convolution just for example on locations with the label 1, the second for label 2, and so on so forth.</p>
<p>ps: Please consider that I receive the locations from another network, so I need the gradient not to be detached for the location vectors.<br>
Any help from the entire community is highly appreciated (surprisingly with a snippet of code)</p>
Can I use Linux-computers connected to the internet to use DistributedDataParallel?

username: Sutter_Productions_E
cooked: <p>Good Day</p>
<p>As example:<br>
I have two computers which both work on Linux and connect both with a ethernet cable to the router. I checked and proved with a simple python script with socket that the computers are reachable. (Code: <a href="https://stackoverflow.com/questions/42415207/send-receive-data-with-python-socket" rel="noopener nofollow ugc">Send/receive data with python socket - Stack Overflow</a> )<br>
Can I use this Linux-computers connected to the internet to use DistributedDataParallel (<a href="https://pytorch.org/docs/master/generated/torch.nn.parallel.DistributedDataParallel.html" rel="noopener nofollow ugc">DistributedDataParallel — PyTorch master documentation</a> )?<br>
I searched the web for an answer but there where no clear answer to this question and so I am not sure if I have to create a special network architecture and install some programms first to use DistributedDataParallel.</p>
<p>I am grateful for every answer and suggestion where I can find this answer. Thank you very much for your time.</p>

username: ptrblck
cooked: <p>I would assume you could use <code>DDP</code> as long as both nodes can reach each other.<br>
However, if the traffic is indeed using an internet connection (and not a local network) I would assume to see a massive slowdown in your training as this would most likely be the bottleneck of your application.</p>

username: Sutter_Productions_E
cooked: <p>Thank you very much for your answer.<br>
So it would become faster if I use a ethernet switch?<br>
I tried this out with two windows computer. They connected but as soon as they did I got an “RuntimeError: Stop_waiting response is expected” generated from “init_process_group”. Do you know why thats happening?</p>
Unexpected output when using src_key_padding_mask

username: sachinruk
cooked: <p>Sorry for the crosspost (should have posted here) but got any thoughts on this?</p><aside class="onebox stackexchange" data-onebox-src="https://stackoverflow.com/questions/69205015/unexpected-output-when-using-src-key-padding-mask">
  <header class="source">

      <a href="https://stackoverflow.com/questions/69205015/unexpected-output-when-using-src-key-padding-mask" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/2530674/sachinruk" target="_blank" rel="noopener nofollow ugc">
    <img alt="sachinruk" src="https://www.gravatar.com/avatar/75d910050ccc1bc26073167739158abf?s=128&amp;d=identicon&amp;r=PG" class="thumbnail onebox-avatar" width="128" height="128">
  </a>

<h4>
  <a href="https://stackoverflow.com/questions/69205015/unexpected-output-when-using-src-key-padding-mask" target="_blank" rel="noopener nofollow ugc">Unexpected output when using src_key_padding_mask</a>
</h4>

<div class="tags">
  <strong>pytorch</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/2530674/sachinruk" target="_blank" rel="noopener nofollow ugc">
    sachinruk
  </a>
  on <a href="https://stackoverflow.com/questions/69205015/unexpected-output-when-using-src-key-padding-mask" target="_blank" rel="noopener nofollow ugc">08:35AM - 16 Sep 21 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
How do I delete persistent workers?

username: thavens
cooked: <p>For some reason, I can’t train for multiple epochs without cv2 or hd5py getting deadlocked. (freezes immediately at the start of the second epoch)<br>
I found that using persistent workers solves the issue.<br>
But after I’m done training how do I go about deleting the extra processes?</p>
<p>using <code>del trainloader</code> doesn’t end the extra processes</p>
Conv2D not working correctly in C++

username: soulslicer
cooked: <p>I have a kernel of size [3,3] and an image of size [B, 3, H, W]</p>
<p>I want to apply the same kernel in parallel across those 3 channels. They do not share data across channels</p>
<p>This is my code</p>
<pre><code class="lang-auto">  auto module = torch::nn::Conv2d(
    torch::nn::Conv2dOptions(3, 3, {3, 3}).padding(1).bias(false));
  module-&gt;weight = kernel.unsqueeze(0) 
                     .unsqueeze(0)
                     .repeat({1, 3, 1, 1})
                     .to(in_image.device());

  return module(in_image);
</code></pre>
<p>The size of the output is [1, 1, H, W]</p>
<p>But the same code implemented in Python</p>
<pre><code class="lang-auto">    conv = torch.nn.Conv2d(in_channels=3, out_channels=3, groups=1, kernel_size= (3,3), padding=3//2, bias=False)
    conv.weights = x_kernel.unsqueeze(0).unsqueeze(0).repeat(1,3,1,1)
    out = conv(rgb_image)
</code></pre>
<p>It gives [1, 3, H, W]. What gives?</p>

username: ptrblck
cooked: <p>Your Python code is wrong, as you are creating a new <code>.weights</code> attribute while you would like to replace the <code>.weight</code> parameter.<br>
Change it to:</p>
<pre><code class="lang-python">x_kernel = torch.randn(3, 3)
rgb_image = torch.randn(1, 3, 24, 24)

conv = torch.nn.Conv2d(in_channels=3, out_channels=3, groups=1, kernel_size= (3,3), padding=3//2, bias=False)
conv.weight = nn.Parameter(x_kernel.unsqueeze(0).unsqueeze(0).repeat(1,3,1,1))
out = conv(rgb_image)
print(out.shape)
&gt; torch.Size([1, 1, 24, 24])
</code></pre>
<p>and you will get the same output size as in <code>libtorch</code>.</p>
Read c++ libtorch tensor into python

username: jloverich
cooked: <p>I am able to write a tensor to a file in c++</p>
<pre><code class="lang-auto">somefunc(at::Tensor&amp; image) {
torch::save(image, "original_image.pt");
</code></pre>
<p>and then when I load into python and look at it’s contents it see</p>
<pre><code class="lang-auto">cpp_tensor=torch.load("original_image.pt")
cpp_tensor
</code></pre>
<p>I get</p>
<pre><code class="lang-auto">RecursiveScriptModule(original_name=Module)
</code></pre>
<p>Am I able to access the tensor from the “RecursiveScriptModule”? When I do the same thing with pytorch it just returns the original tensor.  How do access the original tensor?</p>

username: ptrblck
cooked: <p>This should work to save a tensor in <code>libtorch</code> and load it in Python:</p>
<pre><code class="lang-auto">#include &lt;torch/torch.h&gt;
#include &lt;iostream&gt;

int main() {
  auto tensor = torch::ones({3, 3});
  auto bytes = torch::pickle_save(tensor);
  std::ofstream fout("lala.pt", std::ios::out | std::ios::binary);
  fout.write(bytes.data(), bytes.size());
  fout.close();
}
</code></pre>
<pre><code class="lang-python">&gt;&gt;&gt; import torch
&gt;&gt;&gt; x = torch.load('lala.pt')
&gt;&gt;&gt; x
tensor([[1., 1., 1.],
        [1., 1., 1.],
        [1., 1., 1.]])
</code></pre>
Custom filter, cuda, dataparallel problem

username: 439327bcc62cd8e45e96
cooked: <p>I built a custom filter in pytorch as shown below.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/5/f5f0081658ad1450925d6142e38b745aa9b77daa.png" data-download-href="https://discuss.pytorch.org/uploads/default/f5f0081658ad1450925d6142e38b745aa9b77daa" title="Screen Shot 2021-09-17 at 5.44.56 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/5/f5f0081658ad1450925d6142e38b745aa9b77daa_2_690x146.png" alt="Screen Shot 2021-09-17 at 5.44.56 PM" data-base62-sha1="z5FfdYFgcnT62NprLqFwS5eQTYC" width="690" height="146" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/5/f5f0081658ad1450925d6142e38b745aa9b77daa_2_690x146.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/5/f5f0081658ad1450925d6142e38b745aa9b77daa_2_1035x219.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/5/f5f0081658ad1450925d6142e38b745aa9b77daa_2_1380x292.png 2x" data-dominant-color="393E47"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2021-09-17 at 5.44.56 PM</span><span class="informations">2482×526 65.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>When using only one cpu or gpu, the code works fine, but when using dataparallel and model dataparallel, the code does not run.</p>
<p>Is there any way to solve the dataparallel problem in custom filter?<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/5/c503249e019c860d7bc1e222f5230615e9c0b400.png" data-download-href="https://discuss.pytorch.org/uploads/default/c503249e019c860d7bc1e222f5230615e9c0b400" title="Screen Shot 2021-09-17 at 5.48.31 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/5/c503249e019c860d7bc1e222f5230615e9c0b400_2_690x251.png" alt="Screen Shot 2021-09-17 at 5.48.31 PM" data-base62-sha1="s6QNkiexd355zUOiHYd956kB4fS" width="690" height="251" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/5/c503249e019c860d7bc1e222f5230615e9c0b400_2_690x251.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/5/c503249e019c860d7bc1e222f5230615e9c0b400_2_1035x376.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/5/c503249e019c860d7bc1e222f5230615e9c0b400_2_1380x502.png 2x" data-dominant-color="3C414A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2021-09-17 at 5.48.31 PM</span><span class="informations">2726×992 333 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>Could you post a minimal, executable code snippet which would reproduce the issue so that we could take a look at it, please?</p>
How to run conventional for loop on GPU?

username: Md_Ashikur_Rahman
cooked: <p>Hi,</p>
<p>I would like to know how to run conventional for loop on GPUs? I have 2 GPUs. Below is a sample code for your consideration. It takes around a few minutes to run the code on the CPU. However, I need to run the code within a few sections. Could you please help me out?</p>
<pre><code>maxiter = 10000
for iteration in range(maxiter):
    print(str(iteration) + '--' + str(datetime.datetime.now()))</code></pre>

username: arya47
cooked: <p>Not relevant to PyTorch</p>
<p>Have a look at</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://developer.nvidia.com/how-to-cuda-python">
  <header class="source">
      <img src="https://developer.nvidia.com/sites/all/themes/devzone_new/favicon.ico" class="site-icon" width="48" height="48">

      <a href="https://developer.nvidia.com/how-to-cuda-python" target="_blank" rel="noopener nofollow ugc" title="12:38PM - 19 November 2013">NVIDIA Developer – 19 Nov 13</a>
  </header>

  <article class="onebox-body">
    

<h3><a href="https://developer.nvidia.com/how-to-cuda-python" target="_blank" rel="noopener nofollow ugc">GPU Accelerated Computing with Python</a></h3>

  <p>GPU-Accelerated Computing with Python NVIDIA’s CUDA Python provides a driver and runtime API for existing toolkits and libraries to simplify GPU-based accelerated processing. Python is one of the most popular programming languages for science,...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Can a bunch of frames be an input to a neural network?

username: Wartynas
cooked: <p>Hello, I am trying to predict whether a brain tumor has a specific genetic variant (MGMT promoter) from the MRI scans of the brain. I will try to be explicit and give enough context on the problem. There are 585 patients in the training dataset, each containing 4 folders of different MRI modes: T1w, T2w, T1wCE and FLAIR. Inside each of the 4 folders, there are hundreds of dicom files. Also, each patient is assigned a target value of either 0 (brain tumor doesn’t have MGMT promoter) or 1 (there is MGMT promoter in a tumor).</p>
<p>I have written a custom Dataset for data handling but I am not sure what should be the input to a neural network. Specifically, I would like to treat a group of dicoms from a single mode (for example, FLAIR) as a single input, but then the shape of the resulting numpt array is (number_of_dicoms, 224,224) as the images are grayscaled. Is it possible to feed a tensor of such shape to a neural network, or should I feed each dicom separately (e.g. 224x224 tensor)?</p>

username: arya47
cooked: <p>Have you look at 3D convolutional neural networks, there are multiple notebooks in this Kaggle competition that show how this can be done. There are also multiple discussion posts on the same</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://www.kaggle.com/c/rsna-miccai-brain-tumor-radiogenomic-classification/discussion/270535#1504038">
  <header class="source">
      <img src="https://www.kaggle.com/static/images/favicon.ico" class="site-icon" width="32" height="32">

      <a href="https://www.kaggle.com/c/rsna-miccai-brain-tumor-radiogenomic-classification/discussion/270535#1504038" target="_blank" rel="noopener nofollow ugc">kaggle.com</a>
  </header>

  <article class="onebox-body">
    <img src="https://storage.googleapis.com/kaggle-competitions/kaggle/29653/logos/thumb76_76.png?t=2021-07-28-19-43-35" class="thumbnail onebox-avatar" width="240" height="240">

<h3><a href="https://www.kaggle.com/c/rsna-miccai-brain-tumor-radiogenomic-classification/discussion/270535#1504038" target="_blank" rel="noopener nofollow ugc">RSNA-MICCAI Brain Tumor Radiogenomic Classification</a></h3>

  <p>Predict the status of a genetic biomarker important for brain cancer treatment</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Why is reindexing a Dataset object causing CUDA memory errors

username: Trigaten
cooked: <p>I have a pretty simple Dataset object which loads short videos (400 frames about 512*780 pixels each) from memory.<br>
I have the following lines of code. Notice I index my global_dataset object twice at the same index. Unless I remove the latter indexing operation (last line of code), I get a CUDA memory error regardless of batch size (always 1 for this application) and epochs of training on a small recurrent CNN model. Why might this be? My Dataset object uses <a href="http://torchvision.IO" rel="noopener nofollow ugc">torchvision.IO</a> to load the videos and is very similar to the ones in official Pytorch tutorials. Please let me know if more information would be useful.</p>
<pre><code class="lang-auto">with torch.no_grad():
        one_left_out, label = global_dataset[index]
 
        logits = net(one_left_out)
        
        last = logits[len(logits)-1]
        last = last.unsqueeze(0)
        label = label.unsqueeze(0)
        
        loss = loss_func(last, label)
 
        _, exp = global_dataset[index]

</code></pre>

username: arya47
cooked: <p>Can you share the error and maybe have a look at how much memory is used without the last line ( using nvidia-smi)</p>
Is autograd storing the activations somewhere?

username: aliutkus
cooked: <p>out of curiosity, I would like to know:</p>
<p>when I apply some module to some batch, does autograd actually store the activations of all the layers somewhere ?</p>
<p>It’s not really whether I would like to access them, but I would like to know</p>
<p>thanks, cheers</p>

username: tom
cooked: <p>It depends: for each operation (there might be several in a layer), autograd stores the inputs and/or outputs needed to compute the backward.<br>
On newer PyTorch, the Node objects (.gradfn /. gradfn.next_functions) have “non-official” ._saved… attributes where you can see what has been stored.</p>
<p>Best regards</p>
<p>Thomas</p>

username: aliutkus
cooked: <p>thanks<br>
let’s say we have some classical CNN, with each layer being the application of some conv relu batchnorm thing.</p>
<p>In that case, there would be a tensor of activations that is saved at each layer for batchprop, or  is some smarter thing that is done to save memory ?</p>
<p>These are all pretty stupid questions, but it turns out I can’t readily find information on that topic</p>

username: tom
cooked: <p>Each op decides what it wants for the backward.<br>
(There is a tools/autograd/derivatives.yaml in the source with the definitions.) No particular smartness is applied by default, but people use a technique called checkpointing to save memory (by redoing parts of the compitation).</p>

username: aliutkus
cooked: <p>thanks for the answers</p>
Could NOT find CUDNN (missing: CUDNN_LIBRARY_PATH CUDNN_INCLUDE_PATH)

username: NullPointer
cooked: <p>In the past, I’ve used PyTorch with Python, but I’m looking for better performance in CPP. I have CUDNN installed but the paths I have here aren’t working. Any advice would be greatly appreciated. Thank you</p>
<pre><code class="lang-auto">-- Caffe2: CUDA detected: 11.4
-- Caffe2: CUDA nvcc is: /usr/local/cuda-11.4/bin/nvcc
-- Caffe2: CUDA toolkit directory: /usr/local/cuda-11.4
-- Caffe2: Header version is: 11.4
-- Could NOT find CUDNN (missing: CUDNN_LIBRARY_PATH CUDNN_INCLUDE_PATH) 
CMake Warning at /home/username/Downloads/libtorch/share/cmake/Caffe2/public/cuda.cmake:109 (message):
  Caffe2: Cannot find cuDNN library.  Turning the option off
Call Stack (most recent call first):
  /home/username/Downloads/libtorch/share/cmake/Caffe2/Caffe2Config.cmake:88 (include)
  /home/username/Downloads/libtorch/share/cmake/Torch/TorchConfig.cmake:68 (find_package)
  CMakeLists.txt:7 (find_package)
</code></pre>
<p>CMakeLists.txt</p>
<pre><code class="lang-auto">cmake_minimum_required(VERSION 3.2)

project(apresearch)

set(CMAKE_PREFIX_PATH /home/username/Downloads/libtorch)

find_package(Torch REQUIRED)

set(CUDNN_INCLUDE_DIR /usr/lib/cuda/include)

set(CUDNN_INCLUDE_PATH /usr/lib/cuda/include)

set(CUDNN_LIB_DIR /usr/lib/cuda/lib64)

set(CUDNN_PATH /usr/lib/cuda/lib64/libcudnn.so)

set(CUDNN_LIBRARY_PATH /usr/lib/cuda/lib64)

set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${TORCH_CXX_FLAGS}")

add_executable(${PROJECT_NAME} main.cpp)

target_link_libraries(${PROJECT_NAME} "${TORCH_LIBRARIES}")

set_property(TARGET ${PROJECT_NAME} PROPERTY CXX_STANDARD 14)
</code></pre>

username: NullPointer
cooked: <p>I reinstalled cudnn and it worked!</p>
Get autograd name fails

username: MartinPerry
cooked: <p>In C++, I want to get name of autograd function for tensor. I use this code:</p>
<pre><code class="lang-auto">void TensorPrintData(const torch::Tensor&amp; t){
	std::cout &lt;&lt; t &lt;&lt; std::endl;

	if (const auto&amp; g = t.grad_fn()){
		//crash on gName at the end ob block - assertion failure
		std::string gName = g-&gt;name();
		std::cout &lt;&lt; "Grad: " &lt;&lt; gName &lt;&lt; std::endl;
	}
	else if (t.requires_grad()){
		std::cout &lt;&lt; " &gt; requires_grad" &lt;&lt; std::endl;
	}
}
</code></pre>
<p>but <code>gName</code> causes crash when block ends with assertion failure in Visual Studio 2019 C++. I have Torch 1.9.0 cpu, x64, C++17.</p>
Correct Usage of LayerNorm

username: benihime91
cooked: <p>Hi I want to use LayerNorm to apply per channel normalization for an image.</p>
<p>Assume a minibatch is of shape <em><code>[B, C, H, W]</code></em>, I want to normalize across the <code>C</code> dimension. How to achive this use <code>nn.LayerNorm</code> class ?</p>
ValueError: Target size (torch.Size([128])) must be the same as input size (torch.Size([112]))

username: CasellaJr
cooked: <p>I have a training function, in which inside there are two vectors:</p>
<pre><code class="lang-auto">d_labels_a = torch.zeros(128)
d_labels_b = torch.ones(128)
</code></pre>
<p>Then I have these features:</p>
<pre><code class="lang-auto"># Compute output
features_a = nets[0](input_a)
features_b = nets[1](input_b)
features_c = nets[2](inputs)
</code></pre>
<p>And then a domain classifier (nets[4]) makes predictions:</p>
<pre><code class="lang-auto">d_pred_a = torch.squeeze(nets[4](features_a))
d_pred_b = torch.squeeze(nets[4](features_b))
d_pred_a = d_pred_a.float()
d_pred_b = d_pred_b.float()
print(d_pred_a.shape)
</code></pre>
<p>The error raises in the loss function: `            pred_a = torch.squeeze(nets<a>3</a>)<br>
pred_b = torch.squeeze(nets<a>3</a>)<br>
pred_c = torch.squeeze(nets<a>3</a>)</p>
<pre><code>        loss = criterion(pred_a, labels_a) + criterion(pred_b, labels_b) + criterion(pred_c, labels) + d_criterion(d_pred_a, d_labels_a) + d_criterion(d_pred_b, d_labels_b)
</code></pre>
<p><code>The problem is that d_pred_a/b is different from d_labels_a/b, but only after a certain point. Indeed, when I print the shape of d_pred_a/b it is</code>torch.Size([128])<code>but then it changes to</code>torch.Size([112])` independently.<br>
<a href="https://i.stack.imgur.com/1PdxK.png" rel="noopener nofollow ugc"><img src="https://i.stack.imgur.com/1PdxK.png" alt="error" width="690" height="388"></a></p>
<p>It comes from here:</p>
<pre><code class="lang-auto"># Compute output
features_a = nets[0](input_a)
features_b = nets[1](input_b)
features_c = nets[2](inputs)
</code></pre>
<p>because if I print the shape of features_a is <code>torch.Size([128, 2048])</code> but it changes into <code>torch.Size([112, 2048])</code><br>
nets[0] is a VGG, like this:</p>
<pre><code class="lang-auto">class VGG16(nn.Module):

  def __init__(self, input_size, batch_norm=False):
    super(VGG16, self).__init__()

    self.in_channels,self.in_width,self.in_height = input_size

    self.block_1 = VGGBlock(self.in_channels,64,batch_norm=batch_norm)
    self.block_2 = VGGBlock(64, 128,batch_norm=batch_norm)
    self.block_3 = VGGBlock(128, 256,batch_norm=batch_norm)
    self.block_4 = VGGBlock(256,512,batch_norm=batch_norm)


  @property
  def input_size(self):
      return self.in_channels,self.in_width,self.in_height

  def forward(self, x):

    x = self.block_1(x)
    x = self.block_2(x)
    x = self.block_3(x)
    x = self.block_4(x)
    # x = self.avgpool(x)
    x = torch.flatten(x,1)

    return x
</code></pre>

username: CasellaJr
cooked: <p>I solved. The problem was the last batch. I used <code>drop_last=True</code> in the dataloader and It worked.</p>
Parameter optimization of a pytorch Deep Learning model

username: Sandeep_Kumar_GITAM
cooked: <p>Hello friends,<br>
I have designed a deep learning model using the PyTorch framework for image classification. I know to use Keras tuner to optimize a deep learning model designed using Keras. Can anyone help me in optimizing the hyperparameters of a deep learning model designed using the Pytorch framework? I have two options</p>
<ol>
<li>Convert the PyTorch model to a Keras model and use a Keras tuner on it.</li>
<li>Use any hyperparameter optimization method on the PyTorch model.<br>
Can anyone suggest me and help me with this problem of hyperparameter optimization.</li>
</ol>

username: Samuel_Bachorik
cooked: <p>HI,<br>
you can use</p>
<pre><code class="lang-auto">model= Model()
optimizer = torch.optim.Adam(model.parameters(), lr=0.0001) #ADAM
or
optimizer = torch.optim.SGD(model.parameters(), lr=0.0001) # SGD
</code></pre>
<p>Then you can call</p>
<pre><code class="lang-auto">optimizer.step()
</code></pre>

username: AlphaBetaGamma96
cooked: <p>When looking into Hyperparameter optimization you could have a look into “AutoML” techniques which automate the optimization of hyperparamters. One example within PyTorch is Auto-PyTorch, you can find their github repo here: <a href="https://github.com/automl/Auto-PyTorch" class="inline-onebox" rel="noopener nofollow ugc">GitHub - automl/Auto-PyTorch: Automatic architecture search and hyperparameter optimization for PyTorch</a> !</p>
Is there any difference between x[:, None] and x.unsqueeze(1)?

username: AlphaBetaGamma96
cooked: <p>Hi All,</p>
<p>I’ve got a quick question about reshaping <code>Tensor</code>s. In order to broadcast Tensors I’ve been using <code>x[:, None]</code> to add a new <code>dim</code> to my Tensors. However, this seems to be the same as <code>unsqueeze(1)</code>. Is this technically true, or is it slightly different in any way? For example, is it slower or quicker?</p>
<p>Thank you!</p>

username: KFrank
cooked: <p>Hi Alpha!</p>
<aside class="quote no-group" data-username="AlphaBetaGamma96" data-post="1" data-topic="132300" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/3da27b/48.png" class="avatar"> AlphaBetaGamma96:</div>
<blockquote>
<p>this seems to be the same as <code>unsqueeze(1)</code>.</p>
</blockquote>
</aside>
<p>Autgrad indicates that pytorch is smart enough to convert <code>x[:, None]</code><br>
to <code>x.unsqueeze (1)</code>:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.9.0'
&gt;&gt;&gt; x = torch.ones (2, 3, requires_grad = True)
&gt;&gt;&gt; x.unsqueeze (1)
tensor([[[1., 1., 1.]],

        [[1., 1., 1.]]], grad_fn=&lt;UnsqueezeBackward0&gt;)
&gt;&gt;&gt; x[:, None]
tensor([[[1., 1., 1.]],

        [[1., 1., 1.]]], grad_fn=&lt;UnsqueezeBackward0&gt;)
</code></pre>
<p>But this seems to be something of a special case.  Consider:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; x[:, None, :]
tensor([[[1., 1., 1.]],

        [[1., 1., 1.]]], grad_fn=&lt;SliceBackward&gt;)
&gt;&gt;&gt; x[None, :]
tensor([[[1., 1., 1.],
         [1., 1., 1.]]], grad_fn=&lt;SliceBackward&gt;)
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

username: AlphaBetaGamma96
cooked: <p>Hi <a class="mention" href="/u/kfrank">@KFrank</a>!</p>
<p>Thanks for the response! So, it seems that it should be ok when using it to unsqueeze new dims at the end dimensions but not any in between? (this is the special use case of it?) For example, <code>x[:, None]</code> is fine by <code>x[:, None, :]</code> isn’t?</p>
<p>I’m only using it at the moment in order to broadcast a batch of scalars over a batch of matrices. For example,</p>
<pre><code class="lang-auto">A = torch.randn(100, 2, 4, 4) #some matrcies
scalars = torch.randn(100, 2) #scalar scalars
scaled_A = scalar[:,:,None,None] * A
</code></pre>
<p>Perhaps it might be safer to do this?</p>
<pre><code class="lang-auto">A = torch.randn(100, 2, 4, 4) #some matrcies
scalars = torch.randn(100, 2) #scalar scalars
scaled_A = scalar.unsqueeze(2).unsqueeze(3) * A
</code></pre>
<p>Thank you for the help!</p>
How to select elements from a tensor according to a 2d index?

username: 07hyx06
cooked: <p>Suppose A is a 2D tensor with shape (3,4). B is a 2D tensor with shape (k,2) specify k selected index. The output is a tensor C with shape (k) described as:</p>
<pre><code class="lang-auto">C = []
for i in range(len(B)):
    C.append(A[B[i][0]][B[i][1]])
C = torch.tensor(C)
</code></pre>
<p>I wonder if there exists some Pytorch API equivalent to the above code segment?</p>

username: KFrank
cooked: <p>Hi 07hyx06!</p>
<aside class="quote no-group" data-username="07hyx06" data-post="1" data-topic="132280" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/07hyx06/48/17282_2.png" class="avatar"> 07hyx06:</div>
<blockquote>
<pre><code class="lang-auto">C = []
for i in range(len(B)):
    C.append(A[B[i][0]][B[i][1]])
C = torch.tensor(C)
</code></pre>
<p>I wonder if there exists some Pytorch API equivalent to the above code segment?</p>
</blockquote>
</aside>
<p>Yes:</p>
<pre data-code-wrap="python"><code class="lang-python">A[B[:, 0], B[:, 1]]
</code></pre>
<p>Best.</p>
<p>K. Frank</p>
SROCC metric for image quality assessment

username: Ella
cooked: <p>My question doesn’t refer straightforward to the PyTorch library, but vision problem.<br>
I try to assess image quality with the Spearman rank-order coefficient metric(SROCC). The loss function combines few losses and my model train on two labels (input itself and additional label extracted from original image). So I wonder how I am supposed to assess the quality of the result when I have a combination of two different labels that influence the result. So I thought to separately measure with SROCC metric between output and first label (the image itself) and between output and the second label, then to average both results.<br>
I am not sure about the correctness of my solution, any other suggestions?</p>
Should LSTM/RNN state be kept when inferencing?

username: Long_Pu
cooked: <p>Hi,<br>
Say I am doing time series prediction which predict some value for next time step with past T inputs from historical inputs. Say I am using a RNN module like LSTM or GRU.</p>
<p>In trainning/validation, I’ll feed RNN module with batch of shape (batch_size, T, *) data to get a model.</p>
<p>When inferencing, I can either:</p>
<ol>
<li>Always use past T inputs to get next step prediction, then discard the state of the RNN module. That is: use input from time -T to -1 to get prediction at t=0, then discard the final state of the RNN module and  use input from time -T+1 to 0 to get prediction at t=1 etc.</li>
<li>keep the RNN state, and each time feed only one input to get the prediction. That is:<br>
first use input from time -T to -1 to get prediction at t=1 like above. Then feed the RNN with input at t=1 to get the prediction at t=2 etc.</li>
</ol>
<p>Which one is better? Or It depends on specific problems? Thanks</p>

username: Unity05
cooked: <p>Usually, you use the hidden state of the last layer for each time step t, what corresponds to your second option, I suppose. What is the idea behind option 1?</p>
What is the difference between torch.matmul and torch.bmm?

username: xianqian
cooked: <p>Often times I get confused whether to use <code>torch.matmul</code> or go for <code>torch.bmm</code> I seriously don’t understand why do <code>bmm</code> even exist. In what cases should we use <code>torch.matmul</code> and in what cases should I use <code>torch.bmm</code>?</p>

username: mrityu
cooked: <p>You can always use <code>torch.matmul</code>. <code>torch.bmm</code> is a special case of <code>torch.matmul</code> where both the tensors are 3-dimensional and contains equal number of matrices.</p>
Why same notebook allocating large different vram in two different environment?

username: mobassir94
cooked: <p>you can see this notebook is trainable in kaggle using kaggle’s 16gb vram limit : <a href="https://www.kaggle.com/firefliesqn/g2net-gpu-newbie" class="inline-onebox" rel="noopener nofollow ugc">G2Net GPU Newbie | Kaggle</a></p>
<p>i just tried to run this same notebook locally on rtx3090 gpu where i have torch 1.8 installed and same notebook allocating around 23.3 gb vram,why is this happening and how can i optimize my local environment like kaggle? thanks</p>
Segmentation fault when using torch.profiler

username: fanl
cooked: <p>Hi, everyone. I come across following problem using torch.profiler.</p>
<pre><code class="lang-auto">from torch.profiler import profile, ProfilerActivity

    with profile(activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA]) as prof:
        a = torch.rand(1).cuda()
        a += 1
        print('***Profiling...***) # This line will not be reached if call cuda() on Tensor a.
        train_model(model, datasets, cfg)
    print(prof.key_averages().table(sort_by='cuda_memory_usage'))
</code></pre>
<p>When I run the code above, segmentation fault is raised without any other error information.<br>
If <code>cuda()</code> is not called on Tensor <code>a</code>, the next print function works.</p>
<p>The code works fine without profiling.<br>
I’ll appreciate if anyone point the reasons out.</p>
Mapping between the node in onnx to the pytorch module

username: AlvinZheng
cooked: <p>Hi~ Is there any way to map the nodes in the onnx.graph(exported by torch.onnx.export) to the modules in the pytorch module(for example, conv). The reason I do this is that I need some additional customized attributes in the module to optimize the execution of onnx, but this information is not included in onnx (this information is not used in inference).</p>
Slow batch_gradient

username: Erdi
cooked: <p>I am working on a simple regression problem (X,Y) with ~200K sample size and I tried batch-feeding and no-batch-feeding. It seems when I feed the data in batches, it is significantly slower than just using no-batch at all. I was expecting the otherwise. Here is my code. Can you guys let me know if there is any bug there causing this issue? I attached the major part of the code, thank you!<br>
This is super slow</p>
<p>temp_dataloder = DataLoader(dataset = temp_data, batch_size = 256, shuffle = False)<br>
learning_rate = 0.001<br>
criterion = nn.MSELoss()<br>
optimizer = torch.optim.SGD(model.parameters(), lr = learning_rate)<br>
num_epochs = 100<br>
for epoch in range(num_epochs):<br>
for X, Y in iter(temp_dataloder):<br>
X = X.to(device)<br>
Y = Y.to(device)<br>
y_pred = model(X).to(device)<br>
loss = criterion(y_pred, Y)<br>
loss.backward()<br>
optimizer.step()<br>
optimizer.zero_grad()<br>
if (epoch+1)%10==0:<br>
print(f’epoch : {epoch+1} loss: {loss.item()}’)</p>
<p>And this is super fast<br>
X = temp_data.x<br>
X = X.to(device)<br>
Y = temp_data.y<br>
Y = Y.to(device)<br>
learning_rate = 0.001<br>
criterion = nn.MSELoss()<br>
optimizer = torch.optim.SGD(model.parameters(), lr = learning_rate)<br>
num_epochs = 100<br>
for epoch in range(num_epochs):<br>
y_pred = model(X).to(device)<br>
loss = criterion(y_pred, Y)<br>
loss.backward()<br>
optimizer.step()<br>
optimizer.zero_grad()</p>
<pre><code>if (epoch+1)%10==0:
    print(f'epoch : {epoch+1} loss: {loss.item()}')</code></pre>

username: ptrblck
cooked: <p>It depends on what you are profiling.<br>
The execution of a batch with more than a sample vs. a single sample is expected to be slower, since you are using more operations. However, the epoch time would usually decrease, i.e. the samples per second performance would be faster using a batched training unless your code is bottlenecked by another operation and the actual compute speedup is not visible.</p>

username: Erdi
cooked: <p>Thank you for your prompt response <a class="mention" href="/u/ptrblck">@ptrblck</a> ! I was actually trying to understand if there is any bottleneck in the code. This one is fairly simple example, I printed out execution time of each operation in each batch. It seems It takes ridiculously long time in computation of gradients(~10sec). The very same operation takes a fraction of a second if no batch is used.<br>
I was wondering if you can provide a further insight, thank you!</p>

username: ptrblck
cooked: <p>Based on this description I think you might not synchronize the code properly and your profiling might be wrong. CUDA operations are executed asynchronously so that you would have to synchronize the code via <code>torch.cuda.synchronize()</code> before starting and stopping each timer. Otherwise blocking operations would accumulate the execution times from all previous kernels.</p>
SVD Error "svd_cuda_gesvdj" not implemented for 'Half'

username: okfallah
cooked: <p>Hello,</p>
<p>I am trying to compute the SVD with PyTorch 1.9, using a Nvidia RTX 3080 with NVIDIA-SMI 460.91.03, Driver Version: 460.91.03, and CUDA Version: 11.2.</p>
<p>I get this error when I compute the SVD for a 24x128 matrix:</p>
<pre><code class="lang-auto">  File "/home/kion/simclr-alignment/models/rank_loss.py", line 35, in forward
    U, S, V = torch.linalg.svd(A, full_matrices=True)
RuntimeError: "svd_cuda_gesvdj" not implemented for 'Half'
</code></pre>
<p>Is this error caused by the matrix dimensions? Error also occurs when full_matrices=False.</p>

username: okfallah
cooked: <p>Note: Just checked and the same error occurs with a 192 x 128 matrix.</p>

username: okfallah
cooked: <p>Turns out this issue was caused by mixed-precision (16-bit values). By casting the matrices into a float first, I was able to fix this issue. Keeping thread for visibility of issue.</p>
Best-practice project with 100% GPU usage?

username: Cow_woC
cooked: <p>Is there a sample best-practice project showing pytorch using 100% GPU during training?</p>
<p>I ask because my own project* has 1% GPU usage and in spite of profiling I can’t figure out why. It would help to have a reference project to compare to.</p>
<p>Kind regards,<br>
Gili</p>
<ul>
<li>My project uses a combination of GRU and Linear layers to predict a time series. I get low GPU usage even if I hard-code the inputs (trying to get all work off the CPU).</li>
</ul>
Change Convolution padding when testing

username: Dali
cooked: <p>Hello!<br>
Is there a way to change the padding of torch::nn::Conv2d objects stored in a torch::nn::Sequential after training so that the test network will allow to get the same size as the input ?<br>
Here is a code snippet:</p>
<p>auto Fast=TestNetwork-&gt;getFastSequential();<br>
size_t Sz=Fast-&gt;size();<br>
size_t cc=0;<br>
for (cc=0;cc&lt;Sz;cc++)<br>
{<br>
if (Fast-&gt;named_children()[cc].key()==std::string(“conv”)+std::to_string(cc))<br>
{ Fast-&gt;named_children()[cc].options.padding=1;}<br>
}</p>
<p>It complains about AnyModule object not having a member named options.</p>
How to deal with <PAD> in a self-defined autoencoder

username: Sirui_Li
cooked: <p>I have a batch of tokens and I have padded them into the same length. How could I ignore the loss of pad token in my auto-encoder?</p>
<p>The encoder and decoder look like:</p>
<pre><code class="lang-auto"> self.encoder = nn.Sequential(
            nn.Linear(200, 100),
            nn.Tanh(),
            nn.Linear(100, 50),
            nn.Tanh(),
            nn.Linear(50, 25)
            )
self.decoder = nn.Sequential(
            nn.Linear(25, 50),
            nn.Tanh(),
            nn.Linear(50, 100),
            nn.Sigmoid()
            )
</code></pre>

username: Unity05
cooked: <p>Hi,<br>
I’m not sure if it would actually make sense to pass padded inputs to your auto - encoder.<br>
The result depends also on the padding values in the input.<br>
Depending on what you’re task is, you might want to consider using something like a LSTM Encoder - Decoder.</p>

username: mmg
cooked: <p>Use this…</p>
<pre><code class="lang-python">criterion = nn.CrossEntropyLoss(ignore_index = PAD_IDX)
</code></pre>
<p>where <code>PAD_IDX</code> is the pad token</p>