akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Seperate training for two sequential network

username: iamsaurabh007
cooked: <p>I want to train a convolutional network with triplet loss for getting accurate embeddings and then feed the embeddings to a dense network and train the dense network with other loss.</p>
<p>Now What should be the best practice to do this in pytorch.<br>
i) Make a common network and freeze different part at a time with each loss for training.</p>
<p>ii) Define two different network blocks and train them with each loss.<br>
And if ii) then while training second block, how should i manage with the first block gradient update, With torch.no_grad() while feeding in first block,  or detach the output of first block.</p>
<p>Or any other idea to deal with this</p>
EOFError in PyTorch loader

username: mbehzad
cooked: <p>I am trying to load a dataset using the pytorch’s data loader. When I run</p>
<pre><code class="lang-auto">for batch_idx, (data, label) in enumerate(loader):
    &lt;rest of the code&gt;
</code></pre>
<p>I get the following error:</p>
<blockquote>
<p><strong>Exception has occurred: EOFError       (note: full exception trace is shown but execution is paused at: )</strong><br>
<strong>Ran out of input</strong><br>
<strong>File “”, line 1, in  (Current frame)</strong></p>
</blockquote>
<p>I don’t know what’s wrong. The same dataset runs elsewhere, and I can also see some values in <code>loader.dataset.data.shape</code> and <code>loader.dataset.label</code> with correct values.</p>
<p>Any help?</p>

username: ptrblck
cooked: <p>The <code>EOFError</code> is usually raised by the data loading. Does the stacktrace show more information which line of code is failing?</p>

username: mbehzad
cooked: <p>There is no more information unfortunately, and no info about which line is causing the error. However, when I debug and try to get the first batch of <code>(data, label)</code> in the for loop, it causes this error.</p>
How to learn only the part of the networks?

username: oasjd7
cooked: <pre><code class="lang-auto"># input x -&gt; modelA -&gt; modelB -&gt; modelC

optimizer_c = optim.SGD(modelC, ...)

modelA = nn.Linear(10, 10)
modelB = nn.Linear(10, 10)
modelC = nn.Linear(10, 10)

# First phase: Train only modelC.
x = torch.randn(1, 10)
out = modelC(modelB(modelA(x)))
loss_c = ce(out, target)

optimizer_c.zero_grad()
loss_c.backward()
optimizer_c.step()
</code></pre>
<p>If I only want to train <code>modelC</code>, is this the correct code ?<br>
No <code>detach()</code> or <code>torch.no_grad()</code> is required ?</p>

username: sio277
cooked: <p>This post might help you.</p>
<aside class="quote quote-modified" data-post="4" data-topic="64390">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/what-the-difference-between-detach-detach-and-with-torch-no-grad-in-our-training-phase/64390/4">What the difference between detach(), detach_(),and with torch.no_grad() in our training phase?</a> 
  </div>
  <blockquote>
    They might yield the same result, but in specific edge cases you might get different and unexpected results: 

An optimizer, which doesn’t hold certain parameters, won’t update them. However other optimizers could still update these parameters, if they are passed.
If you are using weight decay, have updated some parameters in the past and are now freezing them, these parameters could still be updated, if they were passed to the optimizer.

detach() could act in the same way as the second point.

…
  </blockquote>
</aside>
Vmap confusions: what is the output?

username: sprajagopal
cooked: <p>I’m using torch version 1.8.1. I imported vmap:<br>
<code>from torch._vmap_internals import vmap</code></p>
<p>I have read the documentation but I’m still confused about the syntax. First, I’ll write down what my objective is.</p>
<p>I have a matrix. I want to process the columns as a batch. Each column is multiplied by a random scalar. So each column results in an equal size column. Is this possible with <code>vmap</code>?</p>
<p>Please avoid suggesting alternative ways (without <code>vmap</code>) to do the above. This is just an example I chose to understand <code>vmap</code>.</p>
<p>What I assume:</p>
<ul>
<li>The first argument to <code>vmap</code> is a function <code>func</code>. The result is a vectorized function <code>vfunc</code>.</li>
<li>The vectorized function <code>vfunc</code> applies <code>func</code> to each column of the given matrix.</li>
</ul>
<p>What I have tried:</p>
<pre><code class="lang-auto"> def generate_scalars(c):
     scalar = 1 if random() &gt; 0.5 else -1
     res = torch.tensor(c.size())
     res[:] = scalar
     return res.T
</code></pre>
<p>The result of the above is a single column  and not a matrix as I expected. I’m sure I’m misunderstanding some part of this.</p>
Disable Peer2Peer Access in PyTorch

username: ralzq01
cooked: <p>Hi guys,</p>
<p>I wonder whether there is a way to disable multi-GPU Peer2Peer Access in Pytorch.</p>
<p>Thanks!</p>

username: ptrblck
cooked: <p>In case you are using NCCL, you could use <code>NCCL_P2P_DISABLE=1</code>.</p>

username: ralzq01
cooked: <p>Thanks!</p>
<p>Any available solutions for cudaMemcpy (device to device)?</p>

username: ptrblck
cooked: <p>What is your use case? Are you using UVA with <code>cudaMemcpyDeviceToDevice</code>?<br>
If you don’t want to use p2p access, you could copy the data to the host and from the host to another device.</p>

username: ralzq01
cooked: <p>Thanks! That’s just a benchmark testing for research purpose.</p>
Multiplying 3d tensor by another 3d tensor, dimension-wise

username: pedro-mgb
cooked: <p>Hello!</p>
<p>My question is the following:</p>
<p>I have 2 3D tensors, and with one dimension in common.</p>
<p>1st tensor dimensions: a x b x c<br>
2nd tensor dimensions: a x d x e</p>
<p>I want to multiply these two tensors along the dimension they have in common (a), and such that their output will be a 4 d tensor (e.g. dimensions b x c x d x e). Pretty much multiplying each vector (with a elements) of tensor 1 by each vector (with a elements) of tensor 2.</p>
<p>I can’t think of an easy built-in way to accomplish this. I can think of iterating over the dimensions and performing torch.dot, but that does not look very good. Does any one know of a better way?</p>
<p>Thank you for reading.</p>

username: KFrank
cooked: <p>Hi Pedro!</p>
<aside class="quote no-group" data-username="pedro-mgb" data-post="1" data-topic="118598" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pedro-mgb/48/37258_2.png" class="avatar"> pedro-mgb:</div>
<blockquote>
<p>I want to multiply these two tensors along the dimension they have in common (a), and such that their output will be a 4 d tensor (e.g. dimensions b x c x d x e).</p>
</blockquote>
</aside>
<p><a href="https://pytorch.org/docs/1.8.1/generated/torch.einsum.html#torch.einsum" rel="noopener nofollow ugc">torch.einsum</a> is the general-purpose tool for “contracting” various<br>
indices of tensors.  Here is an example:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.7.1'
&gt;&gt;&gt; a = 3
&gt;&gt;&gt; b = 2
&gt;&gt;&gt; c = 5
&gt;&gt;&gt; d = 3
&gt;&gt;&gt; e = 4
&gt;&gt;&gt; t1 = torch.randn ((a, b, c))
&gt;&gt;&gt; t2 = torch.randn ((a, d, e))
&gt;&gt;&gt; tt = torch.einsum ('ijk,ilm -&gt; jklm', t1, t2)
&gt;&gt;&gt; tt.shape
torch.Size([2, 5, 3, 4])
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

username: pedro-mgb
cooked: <p>Thank you, that’s exaclty what I wanted!</p>
For multi-task, mutple instances required for loss?

username: Jachilles
cooked: <p>I am doing a kind of multi-task learning. I need 2 cross etropy losses for targets and prediction logits for two different tasks A and B.</p>
<p>Then in this case, a safe way would be</p>
<pre><code class="lang-auto">criterionA = torch.nn.CrossEtnropyLoss()
criterionB = torch.nn.CrossEtnropyLoss()

loss = criterionA(z_A, t_A) + criterionB(z_B, t_B)
</code></pre>
<p>But I am wondering if it is necessary to create the loss instances two times. Would it be okay to simply create a single instance and reuse it as in below?</p>
<pre><code class="lang-auto">criterion_shared = torch.nn.CrossEtnropyLoss()

loss = criterion_shared(z_A, t_A) + criterion_shared(z_B, t_B)
</code></pre>
<p>Also, I would be very grateful which custom is preferrable in general for an arbitrary loss (not specifically cross entropy).</p>
<p>I assume there is a specific reason why pytorch dev. created the loss function as a class.</p>
<p>Many thanks!</p>
Best way to use sparse tensors / optimize this particular scenario?

username: rad
cooked: <p>Hello,</p>
<p>I am working on a custom NLP architecture, which uses two types of featurization for word-level tokens, as described in <a href="https://arxiv.org/pdf/2004.09936.pdf" rel="noopener nofollow ugc">this</a> paper. One of the two is the “sparse features”, which according to the authors is simply a multi-hot encoding of character-level n-grams, with n up to 5.</p>
<p>I assume the correct representation of the per-token n-gram multi-hot encoding is a very sparse vector of space complexity L^n, where L is the number of different characters in the language, and n is maximum n-gram length, which is roughly 12M entries for up-to-5-grams in English.</p>
<p>This “vector” has to be taken through a Linear(in_features=12M, out_features=d_model). Which takes an enormous amount of memory.</p>
<p>How can this task be optimized with sparse representations, or are there any other ways to optimize the memory needed for this transformation? So far, I have thought about sparsifying the representations themselves, but should I (and is there a way to) optimize the Linear layer itself?</p>
Pytorch equivalent of numpy.ma.compress_rowcols

username: lamhoangtung
cooked: <p>Hi guys. Are there any equivalent function for <code>numpy.ma.compress_rowcols</code> in PyTorch where it can suppress the rows and/or columns of a 2-D array that contain NAN value</p>
<p><a href="https://numpy.org/doc/stable/reference/generated/numpy.ma.compress_rowcols.html#numpy.ma.compress_rowcols" class="onebox" target="_blank" rel="noopener nofollow ugc">https://numpy.org/doc/stable/reference/generated/numpy.ma.compress_rowcols.html#numpy.ma.compress_rowcols</a></p>
<p>Of course I can loop through each dims and check for NAN but that will be very inefficient. Thanks for checking by.</p>

username: ptrblck
cooked: <p>You could create mask tensors using <code>torch.isnan</code>, <code>torch.isinf</code>, and <code>torch.isfinite</code>.<br>
Depending on the desired output, you could check the results in specific dimensions or apply the mask directly.</p>

username: lamhoangtung
cooked: <p>Sure, I will use <code>torch.isnan</code> but the dropping row or cols part seems inefficient if the matrix have huge shape</p>

username: ptrblck
cooked: <p>That’s interesting to know. Do you know what would make this operation inefficient and how numpy’s approach solves it?</p>

username: lamhoangtung
cooked: <p>:3 I’m not really sure. I will do some more test and report the result back soon</p>
Save model during running experiment

username: spadel
cooked: <p>Is there a way to save a model from a running script, when not mentioned in the code? I am running a model since 5 days on GPU and save the model whenever a new highscore in the metric is achieved. Unfortunately, the dataset contains many mislabeled samples. Thus, the score is declining at the moment although from a qualitative perspective the model is still making progress. As a result, the model is not saved anymore. Is there a way to save the current model or do I have to run it again?</p>

username: ptrblck
cooked: <p>I don’t think this is possible in a clean way. However, you could check, if you can find any usable code injection libraries, which might be able to inject the saving code into the running process.<br>
Note that this workflow is quite hacky, but you might be able to avoid rerunning the entire training.</p>
Syntax of an image tensor

username: Merwa_Mehmood
cooked: <p>image = generator(to_tensor(codes))[‘image’]</p>
<p>image.shape<br>
output:torch.Size([2, 3, 1024, 1024])</p>
<p>I cannot understand what [‘image’] means in the above line of code.</p>

username: ptrblck
cooked: <p>It seems the <code>generator</code> object returns a Python <code>dict</code> and <code>['image']</code> is indexing the <code>'image'</code> key in this <code>dict</code>.<br>
You could be able to check all keys via:</p>
<pre><code class="lang-python">out = generator(to_tensor(codes))
print(out.keys())
</code></pre>
<p>or by checking the <code>forward</code> method.</p>
How to model GD without having memory issues

username: Btorch
cooked: <p>I’m trying to model GD (not sgd), means that I want to take all the data forward and only then backward.<br>
No doubt that we will need to split it , and I tried it:</p>
<blockquote>
<pre><code>    for batch_idx, (data, target) in enumerate(train_loader):
       if args.cuda:
          data, target = data.cuda(), target.cuda()
       output = model(data)
       print("One iter is out",output.shape)
       loss = criterion(output, target)
       loss_array.append(loss)
   loss = loss_array.mean()
   loss.backward()
</code></pre>
</blockquote>
<p>As you can see, I split into batchs for the forward  pass, arrange all in a small loss array, and only  then make backward pass, the problem is that I’m getting out of memory issues, and it’s weird because the difference from the normal run is saving a small loss tensor, which is negligable.</p>
<p>I’ll be happy for any Idea or advice.<br>
Thanks!!!</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Btorch" data-post="1" data-topic="118523">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/btorch/48/12833_2.png" class="avatar"> Btorch:</div>
<blockquote>
<p>and it’s weird because the difference from the normal run is saving a small loss tensor, which is negligable.</p>
</blockquote>
</aside>
<p>That’s not the case, since each tensor chunk would also store the entire computation graph, so you would use the same memory in the end.<br>
You could simulate the larger batch size using one of the approaches described in <a href="https://discuss.pytorch.org/t/why-do-we-need-to-set-the-gradients-manually-to-zero-in-pytorch/4903/20">this post</a>.</p>
Are samples with no positive ignored in training Faster R-CNN?

username: feiyangsuo
cooked: <p>I’m training Faster R-CNN on underwater fish detection. I have samples from rather monotonous scenes (only fishes in water). This leads the model to recognize every region proposal as positive in practice (any thing appears will be classified as fish).<br>
I want to add some samples from other scenes (but with no positive) to have the model learn more features. Yet if samples with no positive are ignored in training, this attempt would be useless.</p>
Parallelize Conv1D on text

username: Dewei_Hu
cooked: <p>I have an input with size [B, D, T*L], where B for batch size, D for embedding dimension, T for label space, and L for the length of the input. This means that each label has a unique word embedding.<br>
And I have a Conv1D layer with input-size=D, kernel-size=9, and filter-maps=64.</p>
<p>I now want to do Conv1D on each word embedding with the same Conv1D layer. In order not to induce overlaps between different embedding matrices, one naive way is using a for loop for each target. Is there a faster way that I can do this? (Like, I copy the Conv1D layer T times?)</p>
Confusing results with cross-entropy loss

username: niko_e
cooked: <p>Hello together,<br>
I’m working on a dataset for semantic segmantation. I’m doing some experiments with cross-entropy loss and got some confusing results. I transformed my groundtruth-image to the out-like tensor with the shape:<br>
<code>out = [n, num_class, w, h]</code>.</p>
<p>Then I generate my target tensor with this out-tensor:<br>
<code>target = torch.argmax(out, dim=1)</code></p>
<p>and get tensor with the shape <code>[n, w, h]</code>. Finally, I tried to calculate the cross entropy loss</p>
<pre><code class="lang-auto">criterion = nn.CrossEntropyLoss()
loss = criterion(out, tareget)
</code></pre>
<p>and I got a loss of 2.2. Shouldn’t the loss be 0? And what if I implement this loss in my segmentation network?</p>
<p>Many thanks in advance!</p>

username: KFrank
cooked: <p>Hi Nikolas!</p>
<aside class="quote no-group" data-username="niko_e" data-post="1" data-topic="118416" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/niko_e/48/37196_2.png" class="avatar"> niko_e:</div>
<blockquote>
<p>I got a loss of 2.2. Shouldn’t the loss be 0?</p>
</blockquote>
</aside>
<p>Without knowing the values in your <code>out</code> tensor, it’s hard to know what<br>
the loss should be.</p>
<p>However, please note that the <code>input</code> passed into <code>CrossEntropyLoss</code><br>
(your <code>out</code> – the predictions made by your model) are expected to be<br>
<em>logits</em> – that is raw-score predictions that run from <code>-inf</code> to <code>inf</code>.</p>
<p>A <code>target</code>  with values of <code>0.0</code> and <code>1.0</code> corresponds to a middling-weak<br>
prediction, and you need a highly-certain prediction to get a loss of <code>0.0</code>.</p>
<p>Here is a three-class example (for a single prediction – not an image)<br>
that shows a loss of <code>0.0</code> for a “highly-certain” prediction expressed<br>
as logits as expected by <code>CrossEntropyLoss</code>:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; torch.__version__
'1.7.1'
&gt;&gt;&gt; out = torch.tensor ([[0.0, 1.0, 0.0]])
&gt;&gt;&gt; out_logits = torch.tensor ([[-1.e6, 1.e6, -1.e6]])
&gt;&gt;&gt; torch.softmax (out_logits, 1)
tensor([[0., 1., 0.]])
&gt;&gt;&gt; target = torch.argmax (out, dim = 1)
&gt;&gt;&gt; target
tensor([1])
&gt;&gt;&gt; torch.nn.CrossEntropyLoss() (out, target)
tensor(0.5514)
&gt;&gt;&gt; torch.nn.CrossEntropyLoss() (out_logits, target)
tensor(0.)
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

username: niko_e
cooked: <p>Thank you for your answer!<br>
My mistake was treating the output as probabilities, as the mathematical definition of cross entropy requires. I forgot, however, that PyTorch treats them as outputs that don’t need to be summed to 1 and need to be converted to probabilities first using the softmax function.</p>
About DistributedDataParallel, I am confused

username: lei_jiang
cooked: <p>I have two GPU cards, both 12 G.<br>
During my training, batchsize is 1, the memory consumption of my model is about 7 G when on a single GPU card.<br>
Now I use DDP to run my model on two GPU cards, I expect the memory consumption on 2 GPUs both are 3.5G, but in the fact, both of them are 7 G.<br>
it’s normal? How should I do to achieve that?<br>
Thanks</p>

username: ptrblck
cooked: <p>If a single sample uses approx. 7GB on one GPU, it would be the min. memory usage you could get using DDP, since the GPUs cannot process “half samples”.<br>
Given that both devices use 7GB, I assume that both are processing a sample in your script.</p>
Load a pretrained .pth generative ProGAN model

username: Florentino
cooked: <p>I have a .pth model as title says, how can I input a latent vector in the the model and show a correspond images?<br>
Thanks</p>

username: ptrblck
cooked: <p>Assuming the <code>*.pth</code> file contains the <code>state_dict</code> of the model, you would need to create an instance of the model and load the <code>state_dict</code> afterwards. Once this is done, you can pass an input tensor to the model to get the output:</p>
<pre><code class="lang-python">model = ProGAN()
state_dict = torch.load('PATH_TO_STATE_DICT/file.pth')
model.load_state_dict(state_dict)
</code></pre>
PyTorch Dataloaders in-memory

username: mslovett21
cooked: <p>If my dataset has 28,000 images each of them 224 by 224 pixels, (results in around 350 MB of data)<br>
and my GPU has 12 GB memory. Will PyTorch DataLoader load all the data  in the-memory and keep it there?<br>
<strong>I am trying to understand whether it is possible to pass the whole dataset in-memory to another application.</strong>  Or does PyTorch keeps only part of the data in memory at all times?<br>
I would like to understand how dataset size and GPU or CPU memory size (and possible batch size) can influence how much data is in memory?<br>
Thank you</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="mslovett21" data-post="1" data-topic="118471">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mslovett21/48/37222_2.png" class="avatar"> mslovett21:</div>
<blockquote>
<p>If my dataset has 28,000 images each of them 224 by 224 pixels, (results in around 350 MB of data)<br>
and my GPU has 12 GB memory. Will PyTorch DataLoader load all the data in the-memory and keep it there?</p>
</blockquote>
</aside>
<p>The <code>DataLoader</code> doesn’t define, if the data is loaded to the device or the CPU, which is specified in the <code>Dataset</code>.<br>
Depending on your <code>Dataset</code>, you could either pre-load the dataset onto the GPU (in its <code>__init__</code>) or lazily load the data in the <code>__getitem__</code> and push each batch to the GPU inside the training loop.</p>
<p>Also note that you might have an error in the size calculation.<br>
Assuming you are dealing with 28.000 images in the spatial resolution of <code>224x224</code>, the size would be:</p>
<pre data-code-wrap="python"><code class="lang-python"># grayscale stored as 32bit floats:
28000 *  224 * 224 * 4 / 1024**3
&gt; 5.23 GB

# RGB images stores as 32bit floats:
28000 * 3 * 224 * 224 * 4 / 1024**3
&gt; 15.70 GB
</code></pre>
<p>Given this size, I would recommend to lazily load the data and push each batch to the device as described in the <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html">data loading tutorial</a>.</p>
How to apply conv3d with batch of videos splitted in frames?

username: Davide57
cooked: <p>Good morning, everyone.<br>
I am trying to do video classification on a dataset of videos that I have split into N frames per video. After this subdivision, I put all the videos in a tensor and through a data loader I pass a tensor (batch x frames x channels x height x width) to the network.<br>
The frames are 122x112x3.<br>
I just can’t figure out how I have to build the structure of the network so that N videos with K frames come in and N labels go out.</p>
<p>The following code does not work since I get in output a tensor 16x16x1x1x1 but I want 16x1 tensor:</p>
<pre><code class="lang-auto">model = torch.nn.Sequential(
    nn.Conv3d(min_video_frames, BATCH_SIZE, kernel_size=(
        3, 112, 112), padding=0),
    nn.BatchNorm3d(BATCH_SIZE),
    nn.Conv3d(BATCH_SIZE, 256, kernel_size=(
        3, 112, 112), padding=0),
    nn.BatchNorm3d(256),
    nn.Linear(256, 1)
)

</code></pre>

username: ptrblck
cooked: <p>You would need to flatten the activation before feeding it to the linear layer e.g. via <code>nn.Flatten</code>.<br>
This would return an output tensor in the shape <code>[batch_size, 1]</code>.</p>
Torch not checking for inplace operations when autocast

username: Alexander_Koch
cooked: <p>I have noticed that torch doesn’t throw an error that a variable was modified by an inplace operation when autocast is turned on. Why is that? And would my example with autocast work as expected?</p>
<p>Here is a simple example without autocast. This throws a Runtime error:</p>
<pre><code class="lang-auto">RuntimeError: one of the variables needed for gradient computation has been modified by an inplace
operation: [torch.cuda.FloatTensor [1, 1]], which is output 0 of UnsqueezeBackward0, is at version 4;
expected version 3 instead. Hint: enable anomaly detection to find the operation that failed to compute its
gradient, with torch.autograd.set_detect_anomaly(True).
</code></pre>
<pre><code class="lang-auto">model = torch.nn.Linear(1, 1).cuda()
x = torch.empty(3, 1).cuda()
x[0] = model(torch.zeros(1).cuda())
x[1] = model(x[0])
x[2] = model(x[1])

loss = x.mean()
loss.backward()
</code></pre>
<p>Here is the example with autocast turned on:</p>
<pre><code class="lang-auto">model = torch.nn.Linear(1, 1).cuda()
with torch.cuda.amp.autocast():
    x = torch.empty(3, 1).cuda()
    x[0] = model(torch.zeros(1).cuda())
    x[1] = model(x[0])
    x[2] = model(x[1])

loss = x.mean()
loss.backward()
</code></pre>

username: ptrblck
cooked: <p><code>autocast</code> could apply the transformations to <code>FP16</code> tensors internally and would thus work on a clone of the data (in another data type) and would save you from the error.<br>
Generally, if Autocast isn’t raising an error, your operations should work fine.</p>
How to multiply matrix (batch_size, n1) * (n1,n2,n3))?

username: Taiping
cooked: <p>I want a batch matrix to multiply a 3D matrix, where it works ok when mat2 is 2D</p>
<pre><code class="lang-auto">batch_size = 256
mat1 = torch.randn(batch_size,25)
mat2 = torch.randn(25,36)
torch.matmul(mat1, mat2).shape
</code></pre>
<p>However, when mat2 is 3D,</p>
<pre><code class="lang-auto">batch_size = 256
mat1 = torch.randn(batch_size,25)
mat2 = torch.randn(25,36, 36)
torch.matmul(mat1, mat2).shape
</code></pre>
<p>it causes the following errors: <em>RuntimeError: mat1 and mat2 shapes cannot be multiplied</em>.</p>

username: 111414
cooked: <p>I suppose that you want to multiply mat1 with each 25*36 matrix in mat2 (36 matrices in total, i.e. mat2.shape[-1]), and finally get 36 results.<br>
The following code may work <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:"> :</p>
<pre><code class="lang-auto">import torch

b = 256
mat1 = torch.randn(b, 25)
mat2 = torch.randn(25, 36, 36)
mat2_p = mat2.permute(2, 0, 1)  # move the last dimension (matrix number) to the first
res = torch.matmul(mat1, mat2_p)
res = res.permute(1, 2, 0)  # restore

print(res.shape)
</code></pre>

username: Taiping
cooked: <p>Thank you so much for your quick response <a class="mention" href="/u/111414">@111414</a> .<br>
I am sorry I did not describe it clearly.<br>
I actually want the dim1 of mat1 as weights to integrate mat2 along dim0 of mat2, then I will get a matrix (batch, 36,36).<br>
Like<br>
mat1: batchxn1,<br>
mat2: n1xn2xn3,<br>
then mat1*mat2 → batchxn2xn3.</p>

username: 111414
cooked: <p>I think <a href="https://pytorch.org/docs/stable/generated/torch.einsum.html" class="inline-onebox" rel="noopener nofollow ugc">torch.einsum — PyTorch 1.8.1 documentation</a> works:</p>
<pre><code class="lang-auto">import torch
m1 = torch.randn(256, 25)
m2 = torch.randn(25, 36, 36)
res = torch.einsum('bk, kij-&gt;bij', m1, m2)
print(res.shape)
</code></pre>

username: Taiping
cooked: <p>Thanks a lot <a class="mention" href="/u/111414">@111414</a> .<br>
I also found that swapping the position between mat1 and mat2 also works:</p>
<pre><code class="lang-auto">batch_size = 256
mat1 = torch.randn(25,batch_size)
mat2 = torch.randn(36,36,25)
torch.matmul(mat2, mat1).shape
</code></pre>
LSTM to Bi-LSTM

username: Threynaud
cooked: <p>I was implementing the little part of speech tagger from the <a href="http://pytorch.org/tutorials/beginner/nlp/sequence_models_tutorial.html#example-an-lstm-for-part-of-speech-tagging" rel="nofollow noopener">tutorial</a> and I was wondering how I could transform this class into a Bi-Directional LSTM.<br>
I am completely new to pytorch, I don’t know if there is an easy way to do so.</p>
<p>Thanks a lot!</p>

username: SimonW
cooked: <p>PyTorch RNN classes have a <code>bidirectional</code> keyword flag. So this is quite easy. You should simply change to</p>
<pre><code class="lang-python">class LSTMTagger(nn.Module):

    def __init__(self, embedding_dim, hidden_dim, vocab_size, tagset_size):
        super(LSTMTagger, self).__init__()
        self.hidden_dim = hidden_dim

        self.word_embeddings = nn.Embedding(vocab_size, embedding_dim)

        # The LSTM takes word embeddings as inputs, and outputs hidden states
        # with dimensionality hidden_dim.
        self.lstm = nn.LSTM(embedding_dim, hidden_dim, bidirectional=True)  # &lt;- change here

        # The linear layer that maps from hidden state space to tag space
        self.hidden2tag = nn.Linear(hidden_dim * 2, tagset_size)
        self.hidden = self.init_hidden()

    def init_hidden(self):
        # Before we've done anything, we dont have any hidden state.
        # Refer to the Pytorch documentation to see exactly
        # why they have this dimensionality.
        # The axes semantics are (num_layers * num_directions, minibatch_size, hidden_dim)
        return (autograd.Variable(torch.zeros(2, 1, self.hidden_dim)),   
                autograd.Variable(torch.zeros(2, 1, self.hidden_dim)))    # &lt;- change here: first dim of hidden needs to be doubled

    def forward(self, sentence):
        embeds = self.word_embeddings(sentence)
        lstm_out, self.hidden = self.lstm(
            embeds.view(len(sentence), 1, -1), self.hidden)
        tag_space = self.hidden2tag(lstm_out.view(len(sentence), -1))
        tag_scores = F.log_softmax(tag_space, dim=1)
        return tag_scores
</code></pre>
<p>The two lines I changed are marked by <code># &lt;- change here</code></p>
<p>edit: changed another line I missed. see below for details.</p>

username: Threynaud
cooked: <p>Hello Simon,</p>
<p>Thanks a lot, this is way simpler than I thought, but I get the following error when I run the small example after changing the lines you indicated:</p>
<pre><code class="lang-auto">RuntimeError: size mismatch, m1: [5 x 12], m2: [6 x 3]
</code></pre>
<p>Any idea why?</p>

username: SimonW
cooked: <p>Oh right. I forgot that the outputs also have both directions. Then your <code>self.hidden2tag = nn.Linear(hidden_dim, tagset_size)</code> should become <code>self.hidden2tag = nn.Linear(hidden_dim * 2, tagset_size)</code>.  Hope that this works!</p>

username: Threynaud
cooked: <p>Working like a charm, thank you so much!</p>
Use_deterministic_algorithms

username: hadichko
cooked: <p>Hi, I use pytorch 1.7.1+cu110. I would like to use deterministic algorithm for debugging purposes and to increase reproducibility of my predictions. But when I call the algorithm, PyTorch complains that torch has no such attribute. Is it possible that this attribute was introduced after the version I currently use?<br>
See below:</p>
<p>import torch<br>
torch.use_deterministic_algorithms(True)</p>
<p>module ‘torch’ has no attribute ‘use_deterministic_algorithms’</p>

username: Andrea_Rosasco
cooked: <p>Yes, checking the documentation it looks like it was introduce in PyTorch 1.8. This is the documentation for your release: <a href="https://pytorch.org/docs/1.7.1/notes/randomness.html?highlight=reproducibility" class="inline-onebox" rel="noopener nofollow ugc">Reproducibility — PyTorch 1.7.1 documentation</a></p>
<p>It uses</p>
<blockquote>
<p>torch.set_deterministic()</p>
</blockquote>
<p>to ensure you are running deterministic algorithms when available.</p>

username: hadichko
cooked: <p>Thank you. Very helpful.</p>
Using NVSHMEM in Building Pytorch Operator

username: DanielWang
cooked: <p><strong>Hi, All</strong></p>
<p>I want to build a Pytorch operator by using NVSHMEM.<br>
Is there any way that I can do that? Because when we build a standalone NVSHMEM application written in pure C++ and CUDA C, we need to use <code>nvshmrun -n 2</code> to run that application. While in Pytorch, how could we achieve the same goal?</p>
<p><strong>Thanks!</strong></p>
Weighted/Random samplers with custom datasets

username: 111510
cooked: <p>How to use samplers with custom datasets? I use torch to implements model including bert pretrained model for token classification (NER)<br>
I already use weighted loss, but its not enough to train on my heavily imbalanced data<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/e/5e60da413749ec2b38f424cb4266f1c8603ce8be.jpeg" alt="aaaaaaaaaaaaaa" data-base62-sha1="dsUo2YfX6lOh24wFjhItUsekFIO" width="345" height="248"> (and this is not including O tag which up to 70% of all data)</p>
<p>when i tried to use WeightedRandomSampler or any other sampler from torch it doesn’t work<br>
Some custom implementations <a href="https://github.com/ufoym/imbalanced-dataset-sampler" class="inline-onebox" rel="noopener nofollow ugc">GitHub - ufoym/imbalanced-dataset-sampler: A (PyTorch) imbalanced dataset sampler for oversampling low frequent classes and undersampling high frequent ones.</a> also raise similar errors</p>
<p>only DataLoader with shuffle True or False works normal for me, not any sampler</p>
<p>my CustomDataset looks like this</p>
<pre><code class="lang-auto">class CustomDataset(Dataset):
    def __init__(self, tokenizer, sentences, labels, max_len):
        self.len = len(sentences)
        self.sentences = sentences
        self.labels = labels
        self.tokenizer = tokenizer
        self.max_len = max_len

    def __getitem__(self, index):
        sentence = str(self.sentences[index])
        inputs = self.tokenizer.encode_plus(
            sentence,
            None,
            add_special_tokens=True,
            max_length=self.max_len,
            truncation=True,
            padding="max_length",
            # pad_to_max_length=True,
            return_token_type_ids=True
        )
        ids = inputs['input_ids']
        mask = inputs['attention_mask']
        label = self.labels[index]
        label.extend([0] * MAX_LEN)
        label = label[:MAX_LEN]

        return {
            'ids': torch.tensor(ids, dtype=torch.long),
            'mask': torch.tensor(mask, dtype=torch.long),
            'tags': torch.tensor(label, dtype=torch.long)
        }

    def __len__(self):
        return self.len
</code></pre>
<p>i use sampler</p>
<pre><code class="lang-auto">sampler = WeightedRandomSampler(weights=class_weights, replacement=True, num_samples=len(list(tag2idx.keys())))
</code></pre>
<p>get error</p>
<pre><code class="lang-auto">TypeError: list indices must be integers or slices, not str
</code></pre>

username: 111510
cooked: <p>my bad, just sentings in parametres from config with parentethis, and this was cases error while python interpreter told it was sth with my CustomDataset</p>
Does the shape of pytorch gradient always be the same as input?

username: Arthur-99
cooked: <p>when I do backward() to some non-scalar variables $y$, the shape of result is always the same as input $x$.</p>
<p>Is there any method to get a y-shaped result?</p>
<p>e.g.</p>
<pre><code class="lang-python">y = model(x) # x.shape: (B, 1), y.shape: (B, K)
y.backward(torch.ones_like(y)) 
x.grad.shape == x.shape # (B, 1)

&gt;&gt;&gt; True
</code></pre>
<p>But what I want to get is<br>
$$<br>
\frac{\part y}{\part x} = (\frac{\part y_1}{\part x}, \frac{\part y_2}{\part x}, … ,\frac{\part y_k}{\part x})^T<br>
$$<br>
a  result of shape (B, K).</p>
<p>Now my solution is to write a for loop to get each $\frac{\part y_i}{\part x}$ , but it is too slow, is there any better way?</p>
<pre><code class="lang-python">x_grads = []
for i in range(y.shape[-1]): # K
    x_grad = torch.autograd.grad(y[..., i],
                                 x,
                                 torch.ones_like(y[..., i]),
                                 retain_graph=True)[0]
    x_grads.append(t_grad)

x_grads = torch.cat(x_grads, dim=-1)
</code></pre>

username: KFrank
cooked: <p>Hi Fangyu!</p>
<aside class="quote no-group" data-username="Arthur-99" data-post="1" data-topic="118576" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/arthur-99/48/32872_2.png" class="avatar"> Arthur-99:</div>
<blockquote>
<p>when I do backward() to some non-scalar variables $y$, the shape of result is always the same as input $x$.</p>
<p>Is there any method to get a y-shaped result?</p>
</blockquote>
</aside>
<p>I’m not sure that I understand your exact use case, but the beta-version<br>
<a href="https://pytorch.org/docs/1.8.1/autograd.html#torch.autograd.functional.jacobian" rel="noopener nofollow ugc">torch.autograd.functional.jacobian()</a> might be what you want:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; torch.__version__
'1.7.1'
&gt;&gt;&gt; def my_model (x):
...     return torch.arange (2 * x.numel()).reshape ((2, -1)) * x * x
...
&gt;&gt;&gt; x = torch.tensor ([2.0, 3.0, 5.0])
&gt;&gt;&gt; my_model (x)
tensor([[  0.,   9.,  50.],
        [ 12.,  36., 125.]])
&gt;&gt;&gt; torch.autograd.functional.jacobian (my_model, x)
tensor([[[ 0.,  0.,  0.],
         [ 0.,  6.,  0.],
         [ 0.,  0., 20.]],

        [[12.,  0.,  0.],
         [ 0., 24.,  0.],
         [ 0.,  0., 50.]]])
</code></pre>
<p>Best.</p>
<p>K. Frank</p>
How to implement Time delay Neural Network(TDNN)-LSTM?

username: 111509
cooked: <p>I referred to the TDNN, TDNN-LSTM, TDNN-Attention models provided by Kaldi. I wanted to use this to implement the model with Pytorch, but it was difficult to implement the following:</p>
<ol>
<li>delay : the delay to be used in the recurrence of LSTMs</li>
<li>decay-time : an approximate maximum on how many frames can be remembered via summation into the cell contents</li>
<li>projection-dim : the output dimension of the recurrent-projection-matrix<br>
Has anyone ever had a github repository for reference or implemented such a model? I’d appreciate it if you could show me the implementation code.</li>
</ol>
RuntimeError: Can not iterate over a module list or tuple with a value that does not have a statically determinable length

username: blackberry
cooked: <p>Hello there,</p>
<p>I am trying to compile an SSD-based object detector using <code>torch.jit.script(model)</code>, and I am getting this error message for the following code segment:</p>
<pre><code class="lang-auto"># apply multibox head to source layers
# sources: python list (length: 7)
# self.loc: nn.ModuleList (length: 7)
# self.conf: nn.ModuleList (length: 7)
for (x, l, c) in zip(sources, self.loc, self.conf):
    #~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ &lt;--- HERE
    loc.append(l(x).permute(0, 2, 3, 1).contiguous())
    conf.append(c(x).permute(0, 2, 3, 1).contiguous())
</code></pre>
<p>How can I fix this?</p>
<p>Note: I had to use <code>for k,v in enumerate(module_list): x = v(x)</code> for “expected integer literal for index” error in an earlier segment of the code and it worked. However, this error is different.</p>
<p>Thanks.</p>

username: blackberry
cooked: <p>Solved.</p>
<p>In case someone encounters the same issue, here is how to change the code to be able to compile it:</p>
<pre><code class="lang-auto">for i, (l, c) in enumerate(zip(self.loc, self.conf)):
   x = sources[i]
   loc.append(l(x).permute(0, 2, 3, 1).contiguous())
   conf.append(c(x).permute(0, 2, 3, 1).contiguous())
</code></pre>
How much faster new module fft pytorch then numpy.fft?

username: Vadimcg
cooked: <p>How much faster new module fft pytorch then numpy.fft ?  Сould you give me some exact data(graph)? Something like this: if i use numpy.fft in this task i will spend x time , but if i use pytorch.fft …</p>

username: ptrblck
cooked: <p>You could profile your workload via e.g. <code>torch.utils.benchmark</code> and compare different shapes of the PyTorch implementation vs. the numpy one.</p>
Depth first in 3D Convs?

username: kuonlp
cooked: <p>I noticed that Pytorch <a href="https://pytorch.org/docs/stable/generated/torch.nn.Conv3d.html" rel="noopener nofollow ugc">documentation</a> suggests that the input dimensions of 3D images are (depth, height, width). In contrast, in TF <a href="https://www.tensorflow.org/api_docs/python/tf/keras/layers/Conv3D" rel="noopener nofollow ugc">documentation</a> these dimensions are referred as (dim1, dim2, dim3). Therefore, maybe not but I was wondering if there is any difference (e.g., results, efficiency) in using (depth, height, width) vs. (height, width, depth), since the latter is more natural (at least for me).</p>
<p>Thanks.</p>

username: ptrblck
cooked: <p>PyTorch uses the channels-first layout by default and you could switch to <a href="https://pytorch.org/tutorials/intermediate/memory_format_tutorial.html">channels-last</a>, in case you are using mixed-precision training and your GPU supports TensorCores to speed up these operations. Note that the user-facing shapes would still be shown in the default channels-first layout as described in the tutorial.</p>
Working with Criterion CrossEntropyLoss for multilabel Classification

username: TSDK
cooked: <p>I know this topic has been covered in the past. But I just wanted to make sure that if it is the correct way of what I’m doing. From my understanding of previous post on this:<br>
 → Label should be torch.long() type<br>
–&gt;We should pass the indicies of the label</p>
<pre><code class="lang-auto">    for batch_num, data in enumerate(trainloader, 0):

        # get the images and labels.
        inputs = data["image"].to(device)
        labels = data["label"].to(device)
        labels = labels.long()

        if config.use_amp_grad:

            # forward + backward
            with autocast():
                outputs = net(inputs)
                loss = criterion(outputs, torch.max(labels, 1)[1])
</code></pre>

username: ptrblck
cooked: <p>In case you are using <code>nn.CrossEntropyLoss</code> or <code>nn.NLLLoss</code>, then yes: the targets are expected as <code>LongTensor</code>s containing the class indices in the range <code>[0, nb_classes-1]</code>.</p>
RuntimeError: binary_cross_entropy and BCELoss are unsafe to autocast

username: siheming
cooked: <p>I am using autocast with my model and running into the following error:</p>
<pre><code class="lang-auto">RuntimeError: torch.nn.functional.binary_cross_entropy and torch.nn.BCELoss are unsafe to autocast.
Many models use a sigmoid layer right before the binary cross entropy layer.
In this case, combine the two layers using torch.nn.functional.binary_cross_entropy_with_logits
or torch.nn.BCEWithLogitsLoss.  binary_cross_entropy_with_logits and BCEWithLogits are
safe to autocast.
</code></pre>
<p>I am trying to compare two quite similar models. The first one uses BCEWithLogitsLoss, so no issues there. The second one however does not. The second model uses the softmax function to calculate the probability of the positive example by using the softmax function. It does so by passing the output of the last layer to two dense layers. These are then stacked and the softmax function is applied over the stacked dimension. The pseudo code looks like this:</p>
<pre><code class="lang-auto">class Model(nn.Module):
    def __init__(self):
        self.pos_prob = torch.nn.Linear(100, 1)
        self.neg_prob = torch.nn.Linear(100, 1)
        self.softm = torch.nn.Softmax(dim=0)
        self.loss_fn = torch.nn.BCELoss

    def forward:(self, input, labels):
        probs_pos = self.pos_prob(input)
        probs_neg = self.neg_prob(input)

        probs = self.softm(torch.stack((probs_pos, probs_neg)))[0, :, :].squeeze()
        loss_fn = self.loss_fn()
        loss = loss_fn(probs, labels).view(-1, 1)
        return loss
</code></pre>
<p>is there any way to still use AMP?<br>
Thanks in advance</p>

username: ptrblck
cooked: <p>Note that <code>nn.BCELoss</code> would usually require the usage of <code>sigmoid</code> not <code>softmax</code> as it could be used for a binary or a multi-label classification.<br>
In any case, you could disable <code>autocast</code> using the context manager for this operation.</p>
How to plot psnr Vs Bitrate of an image

username: OuisYasser
cooked: <p>Hello mates, sorry for creating this issue here ( because this community is better than stackoverflow)<br>
So I want to plot the psnr Vs Bitrate graph of an image which is used in compression papers, etc…<br>
If there is any available code in python or matlab that would be helpful or may be some explanation which help to go throw the algorithm.<br>
Thanks<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/7/f7351df26b4e14df99c02a65fd09b66f2e54b541.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/f7351df26b4e14df99c02a65fd09b66f2e54b541" title="Screenshot_20210415-223548__01"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/7/f7351df26b4e14df99c02a65fd09b66f2e54b541_2_690x426.jpeg" alt="Screenshot_20210415-223548__01" data-base62-sha1="zgTJK7TLOsVgCKpehPMYxbqJ64p" width="690" height="426" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/7/f7351df26b4e14df99c02a65fd09b66f2e54b541_2_690x426.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/f/7/f7351df26b4e14df99c02a65fd09b66f2e54b541_2_1035x639.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/7/f7351df26b4e14df99c02a65fd09b66f2e54b541.jpeg 2x" data-dominant-color="F0F0EF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot_20210415-223548__01</span><span class="informations">1080×668 122 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Squeezing the lstm output which is the input of a linear layer

username: ccalafiore
cooked: <p>hi,</p>
<p>I am making a custom model where the output of a lstm is the input of a linear layer. The shape of the last state h_T has shape (1, batch_size, n_features), while the linear model expects an input with shape (batch_size, n_features). Is it correct to squeeze h_t? I am concerned the error may not be backpropagated from the linear layer to the lstm if I squeeze h_T.</p>
<pre><code class="lang-auto">    def forward(self, x):
        lstm_out, (h_T, c_T) = self.lstm(x)
        fc_out = self.fc(h_T.squeeze(0))
        return fc_out
</code></pre>