akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Backward() in custom conv2d

Loud_BoomBox: <p>Hi everyone,</p>
<p>I am a beginner to pytorch and trying to build a custom convolution layer (conv2d) where instead of the default multiplication I replace it with my own multiplication algo. I’ve attached my implementation below.</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/bALAJi-aDItHYa/Custom-conv2d">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/bALAJi-aDItHYa/Custom-conv2d" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/344;"><img src="https://opengraph.githubassets.com/208c2a391d6d30289ca75e280e9995514962fc8205805ae81e1462e716326ec1/bALAJi-aDItHYa/Custom-conv2d" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/bALAJi-aDItHYa/Custom-conv2d" target="_blank" rel="noopener nofollow ugc">GitHub - bALAJi-aDItHYa/Custom-conv2d</a></h3>

  <p>Contribute to bALAJi-aDItHYa/Custom-conv2d development by creating an account on GitHub.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>I am getting the following error during backward pass - <strong>Function MBM_conv2dBackward returned an invalid gradient at index 1 - got [16, 32, 28, 28] but expected shape compatible with [32, 1, 3, 3]</strong>:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/a/7aecadd8b60f90cf553d336678a4d49978b25bca.png" data-download-href="https://discuss.pytorch.org/uploads/default/7aecadd8b60f90cf553d336678a4d49978b25bca" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/a/7aecadd8b60f90cf553d336678a4d49978b25bca.png" alt="image" data-base62-sha1="hxrkHU4vrG3YgJ78jfDWMfv8rr4" width="690" height="273" data-dominant-color="3A3B36"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">854×338 61.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>There seems to be a problem with the implementation of <strong>grad_weight</strong> calculation in the file:- <strong>custom_conv2d.py</strong></p>
<p>My doubts are the following:</p>
<ol>
<li>Where does needs_input_grad variable come from?</li>
<li><strong>input</strong> dimensions → [16,1,28,28] - (batch size = 16; 28x28 images)<br>
<strong>grad_weight</strong> dimensions → [32,1,3,3] - (32 3x3 kernels)<br>
What is the mistake being done during the grad_weight calculation? Or in the backward() implementation in general?</li>
</ol>
<p>I couldn’t figure out the problem. Would really appreciate any help! Thanks!</p>
GPU Memory Usage Handling with try/except

Kabir_Nagrecha: <p>I am writing a function with attempts to find the upper bound of the possible model size. I do this in a loop which at each iteration “tries” to append a new layer to a moduleList, constructs a model off of that list, then attempts a single forward pass.</p>
<p>In the exception, I delete the moduleList, the model, and <em>attempt</em> to delete the output of the forward pass. Because the forward pass failed however, the output of course does not exist. The issue is, whatever intermediate computations were attempted on the way to the failure (mid-forward-pass) persist and eat up memory. How do I free them? Is there a way to kill ALL GPU tensors at this moment?</p>
<p>Example Code (not real):</p>
<pre><code class="lang-auto">modules = []
for i in range(1000):
    try:
        modules.append(nn.Linear(10, 10)).to(device)
        model = nn.Sequential(*modules)
        out = model(input_batch)
        del out
        del model
   except:
        try:
             del out
        except:
             pass
        del model
        del modules
        gc.collect()
        torch.cuda.empty_cache()
        print(torch.cuda.memory_allocated(0)) # this will show that memory is still full!
</code></pre>

Kabir_Nagrecha: <p>Just found this…<a href="https://pytorch.org/docs/stable/notes/faq.html#my-out-of-memory-exception-handler-can-t-allocate-memory" class="inline-onebox" rel="noopener nofollow ugc">Frequently Asked Questions — PyTorch 1.7.0 documentation</a></p>
<p>My code worked this whole time…7-8 hours trying to debug it…and now I found this <img src="https://discuss.pytorch.org/images/emoji/apple/sob.png?v=9" title=":sob:" class="emoji" alt=":sob:"></p>
Image reconstruction performance is not improving

Alok1: <p>Hey guys,<br>
I am working on image reconstruction problem, where the input to the network is speckle pattern image capture by lensless camera. I have followed U-Net architecture + residual connections to maintain the original information of the input image(1, 1, 128, 128). There are four encoders and four decoders, for simplicity I am mentioning one forward function of the encoder and one for the decoder.</p>
<h2><a name="encoder-from-1-4-1" class="anchor" href="#encoder-from-1-4-1"></a>“encoder from 1-4”</h2>
<p>conv2d -&gt;relu-&gt;<br>
batchnorm_1</p>
<p>conv2d-&gt;relu<br>
conv2d_&gt;relu<br>
batchnorm_2<br>
residual connection = batchmorm_1 + batchnorm_2</p>
<h2><a name="decoder-from-1-4-2" class="anchor" href="#decoder-from-1-4-2"></a>“decoder from 1-4”:</h2>
<p>convtranspose2d-&gt;relu_&gt;batchnorm<br>
concatination connection</p>
<p>conv2d -&gt;relu<br>
batchnorn_1</p>
<p>conv2d-&gt;relu<br>
conv2d_&gt;relu<br>
batchnorm_2<br>
residual connection = batchmorm_1 + batchnorm_2</p>
<p>Moreover, I am using ADAM optimizer with lr = le-3, Loss functions that I have tried MSE, MAE, SSIM, BCELoss. However, the learning curve is no more improving, therefore, the model is not able to predict the intensity of each pixel properly thus resulting in blurry and unstructured images. Here is the example image of the reconstruction.<br>
Any suggestion on how to improve the reconstruction ability of the network would be extremely helpful. In case of more clarity on code, please let me know.<br>
Thanx in advance…</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/c/9c3ae0c0a87790bed0565cf7d3d209ba19a4c3ac.png" alt="unstructure" data-base62-sha1="mi4D6LUeynrxWdOIChAHOFnmW8Q" width="585" height="227"></p>
Add graphs of custom computation graph to tensorboard

pecey: <p>I was wondering if there is a way to add a custom computation graph to tensorboard using PyTorch. The  website says it works for PyTorch models and tensors. I didn’t find examples where the graphs are being added to tensorboard for custom computation graphs.</p>
<p>So basically my function takes in a couple of tensors, performs calculations within the function creating more tensors and then outputs a result. I wanted to be able to visualize the interactions between the various tensors.</p>
<p>I tried calling <code>writer.add_graph(my_func, (my_args1, my_args2))</code>, but that resulted in an error stating : <code>'function' object has no attribute 'param_init_net'</code></p>
<p>I have tried to create an extremely simple working example here.<br>
<a href="https://colab.research.google.com/drive/1AIcObBl1QTc3h0hryWo1ZO7ZJ5hk_MfT?usp=sharing" class="onebox" target="_blank" rel="nofollow noopener">https://colab.research.google.com/drive/1AIcObBl1QTc3h0hryWo1ZO7ZJ5hk_MfT?usp=sharing</a></p>
<p>Can someone help me out on this?</p>
<p>Thank you.</p>

ddrobny: <p><a class="mention" href="/u/pecey">@pecey</a> Did you have any luck with this issue?</p>
<p>I am also trying to get a graph visualisation for arbitrary Pytorch code/functions. To me, it seems that <code>add_graph()</code> requires a model as an input. I am using torch for autograd functionality without DL. Is it possible to get a computational graph like in TF?</p>
Inverse of Mode: Least Common Item in Each Column

Kamer_Ali_Yuksel: <p>Hello,</p>
<p>I would like to get the least common item of each column in a matrix.</p>
<p>Can you please let me know in case there exists a convenient way?</p>
<p>Note: I only have two unique values in each column. I want to get<br>
the one with the least occurrence rather than the frequent one.</p>
<p>Sincerely,<br>
Kamer</p>

Kamer_Ali_Yuksel: <p>mode_val = weights.mode(dim=0)[0]<br>
mode_val = mode_val.unsqueeze(0).repeat_interleave(10, dim=0)<br>
inds = (weights != mode_val).float().argmax(dim=0)</p>
<p>Now, I just need to get the values of given indices for each column.<br>
I am trying with index_select, gather and scatter_ but no luck so far.</p>
<p>Basically, I need something that takes a vector of indices and then<br>
select those indices from each column in a matrix.</p>
<p>Note: Without the argmax, there is also a Boolean matrix of same shape to select.<br>
I couldn’t use that to get the values and thus I was trying with indices of each column.</p>
<ul>
<li>I have also tried masked_select with Boolean matrix but that doesn’t seem to do it.</li>
</ul>

Kamer_Ali_Yuksel: <p>This is working but I am not sure if it is correct, as it doesn’t have my expected behavior.<br>
mode_val = weights.mode(dim=0)[0]<br>
mode_val = mode_val.unsqueeze(0).repeat_interleave(2, dim=0)<br>
(weights * (weights != mode_val)).sum(dim=0)</p>
How would I build a deep Q network where each observation is a 2D matrix input?

torch_user_92: <p>Hi how’s it going?</p>
<p>I’m familiar with supervised learning in pytorch where each observation is a row and the input node is just the number of columns of the data point.</p>
<p>However, I’m trying to build a network that takes in a “state” and returns a choice of “action”. The state is how close a bunch of drivers are to the pick up restaurant and to the delivery address, and the action returned should be the best driver for the job given their current location. I will take care of the reward function logic and what not, but I’m just trying to figure out the model architecture in pytorch</p>
<p>Here is an example of what would be one “state” that I would like as an input to the network:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/7/d7caedfb0e40c7a0541b689d5469c2b3d560e0db.png" alt="Screenshot (12)" data-base62-sha1="uMZqI85UH5f2RqPBzeW2kWdPGdl" width="630" height="214"></p>
<p>So every observation would be a 4x2 matrix (excluding name as a feature), and I would like the output layer to return the row index of which driver is best fit for the job. Obviously at first it’ll have no clue but as I build the reward function it will learn that selecting the driver closest to the restaurant and closest to the delivery address lead to maximized reward.</p>
<p>Any help would be greatly appreciated!</p>
<p>Cheers!</p>
Problem with multiplying a batch of sparse tensor with a batch of dense tensors

sayan_ghosal: <p>Hi,</p>
<p>I am facing some problems regarding batch sparse multiplication. I have a batch of sparse graphs, <strong>A ( n x 10000 x 1000).</strong> Which I am storing as list of sparse tensors, <strong>A_s =  [ sparse_tensor(A_1), …]. My input matrix, X has a dimension of [n x 10000 x 10].</strong> Now I want to do batch multiplication, i.e. the first graph gets multiplied with the input from subject 1. To do that I am first putting all my graphs A_i on the diagonals of a big graph graph, and I am manually keeping track of the non-zero indices and non-zero values. Then I create this Big sparse matrix, <strong>A_big</strong> whose diagonals are all those graphs. The dense version of this matrix has size (n<em>10000, n</em>10000).  After, that I am resizing my <strong>X → X_modified ( n*10000 x 10)</strong>. Now, I am performing the <strong>torch.sparse.mm( A_big, X_modified)</strong>. During forward pass everything works fine. But, during backward pass I am getting the error</p>
<p><strong>"RuntimeError: CUDA out of memory. Tried to allocate 2875.53 GiB (GPU 0; 3.94 GB total capacity; 1.31 GiB already allocated; 1.68 GiB free; 1.44 GiB reserved in total by PyTorch)"</strong></p>
<p>However, if I try to do the same this using a <strong>for loop</strong> and performing the sparse multiplication individually on each subject I am <strong>not getting this error.</strong></p>
<p>To give some context, the non-zero values of my sparse graph are learnable using some attention mechanism, and the number of learnable parameters there are 10. Using the sparse implementation I am storing only the non-zero values so I don’t understand why I am getting this error.</p>
<p>Any help is really appreciated!</p>
<p>Thank you!</p>
Implementing cross label dependency loss

Hasan: <p>Hi, I am trying to implement CLD loss as defined in <a href="https://arxiv.org/pdf/1707.00418.pdf" rel="noopener nofollow ugc">this paper</a> (i.e., equation 3) and have two questions:</p>
<ol>
<li>Can someone check if I did the implementation correctly (code attached blow)?</li>
<li>In my implementation, I used the “for loop” to select each row and then process it following equation (3). Is there a way to get rid of this “for loop”?</li>
</ol>
<pre><code class="lang-auto">def cdl(y_hat, y_true):
    """
    implementation of CDL as defined in https://arxiv.org/pdf/1707.00418.pdf
    :param y_hat: model predictions, shape(batch, classes)
    :param y_true: labels (batch, classes)
    :return: loss
    """
    loss = torch.zeros(y_true.size(0))
    for idx, (y, y_h) in enumerate(zip(y_true, y_hat)):
        y_z, y_o = (y == 0).nonzero(), y.nonzero()
        output = torch.exp(torch.sub(y_h[y_z], y_h[y_o][:, None])).sum()
        num_comparisons = y_z.size(0) * y_o.size(0)
        loss[idx] = output.div(num_comparisons)
    return loss.sum()
</code></pre>
<p>Thank you.</p>

Eta_C: <p>If <code>y_true</code> looks like <code>[[1, 1, 0], [0, 1, 1]]</code> and <code>y_hat</code> looks like <code>[[0.1, 0.2, 0.3], [0.4, 0.5, 0.6]]</code>, try the following “no for” implementation. (If your implementation is correct.)</p>
<pre><code class="lang-python">def cdl(y_hat, y_true):
    pre_sub = (y_hat[:, :, None] - y_hat[:, None, :])
    pre_exp = torch.exp(pre_sub)
    y_z, y_o = y_true == 0, y_true != 0
    num_comparisons = y_z.sum(dim=1) * y_true.sum(dim=1)
    pre_exp = pre_exp / num_comparisons[:, None, None]
    mask = torch.logical_and(y_z[:, :, None], y_o[:, None, :])
    loss = pre_exp[mask].sum()
    return loss
</code></pre>
<p>PS: Since <code>pre_sub</code> and <code>pre_exp</code> calculate all pairs, if batch &lt;&lt; classes, maybe “for implementation” is faster(I am not sure)?</p>

Hasan: <p>Thanks for the help, I checked your code and it produced the same results as mine. In my case, batch &gt;&gt; classes for sure.</p>
Error with shortcut layer in yolov3

eslam_fouda: <p>I face the following error with this line<br>
x = outputs[i-1] + outputs[i+from_]<br>
RuntimeError: The size of tensor a (210) must match the size of tensor b (208) at non-singleton dimension 3</p>

ptrblck: <p>Is <code>outputs</code> a <code>list</code> containing tensors with different shapes?<br>
If so, you would have to make sure the shapes are equal on the tensors, which should be added together. Otherwise this shape mismatch error will be raised.</p>

eslam_fouda: <p>I’m beginner. I try to run the model in that link:</p><aside class="onebox githubfolder">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">
      <a href="https://github.com/ayooshkathuria/YOLO_v3_tutorial_from_scratch" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:400/400;"><img src="https://avatars.githubusercontent.com/u/22730263?s=400&amp;amp;v=4" class="thumbnail" width="400" height="400"></div>

<h3><a href="https://github.com/ayooshkathuria/YOLO_v3_tutorial_from_scratch" target="_blank" rel="noopener nofollow ugc">ayooshkathuria/YOLO_v3_tutorial_from_scratch</a></h3>


  <p><span class="label1">Accompanying code for Paperspace tutorial series "How to Implement YOLO v3 Object Detector from Scratch"</span></p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<p>
the output is dictionary caches the outputs for the route layer</p>
To(device) removes tensor from model parameters

benoriol: <p>I have found the following issue when working with the nn.Parameter() class.</p>
<p>Inside a custom nn.Module class, I defined a tensor such as <code>self.trainable_tensor = nn.Parameter(torch.rand(1, 2, 3)).to(device)</code>.</p>
<p>I set the attribute as a nn.Parameter object in order for it to be included in the parameter list and therefore be trained. After many hours of debugging I found out that the to(device) thing actually removes it from this list, only when the device is CUDA. I don’t know why is that, whether it is a bug or what. I understand that It is not necessary to put the .to(device) thing, because if the attribute is already in the parameters list, when you do <code>model.to(device)</code> this tensor will be already part of it and will be moved to the device type. I just supposed that putting that there would be superfluous, but actually, it completely messed my implementation.</p>
<p>Is this a bug? Or is there some concept I am missing? Why does .to(device(“cuda”)) remove the tensor from the parameter list?</p>
<p>Thanks,</p>
<p>Ben</p>

InnovArul: <aside class="quote no-group" data-username="benoriol" data-post="1" data-topic="109808">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/benoriol/48/15004_2.png" class="avatar"> benoriol:</div>
<blockquote>
<p>self.trainable_tensor = nn.Parameter(torch.rand(1, 2, 3)).to(device)</p>
</blockquote>
</aside>
<p>You have to move the tensor before creating the <code>nn.Parameter</code>.<br>
i.e.,</p>
<pre><code class="lang-auto">self.trainable_tensor = nn.Parameter(torch.rand(1, 2, 3).to(device))
</code></pre>
<p>In your case, executing <code>.to(device)</code> after <code>nn.Parameter</code> creation has returned a tensor.<br>
i.e., <code>self.trainable_tensor</code> was assigned with a tensor.</p>
<p>You can print the type of variable (<code>type(self.trainable_tensor)</code>) to make sure that you have created <code>nn.Parameter</code> correctly.</p>

benoriol: <p>Woah thanks, that makes sense!</p>
<p>However, the reason for my long debugging time is that when it was assigned the cpu device, this didn’t happen. Therefore, training with cpu yielded a completely different model that when training with CUDA. MIght that be a bug? I mean, the model should be (almost) the same regardless on what device you train.</p>
<p>Thanks!</p>
<p>Benet</p>

InnovArul: <p><code>.to()</code> checks if the memory content is already in the given device and then, acts to move the memory only if the device is different. In case of <code>cpu</code>, the memory was already in the device that’s requested. So, <code>.to()</code>. did not act on the memory.</p>

benoriol: <p>Oh I see, thanks!</p>
<p>Ben</p>
How to introduce a loss to get two matrices similar

spadel: <p>During my training I predict two covariance matrices, which I want to become similar. I therefore want to introduce a loss term, that forces that. My naive approach was to calculate a loss based on the absolut difference. The problem is, that this also causes the matrices to become small, which is not what I want. Do you guys have ideas here?</p>

ChickenTarm: <p>Couldn’t you have a loss that is the divide by the norm of the matrices? That would make the loss large if the norm small which means your matrix is small.</p>
Help me for this error:torch.nn.modules.module.ModuleAttributeError:

yangyi_hpu: <p>The following code, I hope to add a class “preprocess” for NN. However, I get the error:torch.nn.modules.module.ModuleAttributeError: ‘Darknet’ object has no attribute ‘prePrp’</p>
<pre><code class="lang-auto">import torch.nn as nn
class preprocess(nn.Module):
    def __init__(self):
        super(preprocess, self).__init__()
    def forward(self, x):
        print('here')
        return x + 200
class SuperResolutionNet(nn.Module):
    def __init__(self):
        super(SuperResolutionNet, self).__init__()
        self.relu = nn.ReLU(inplace=False)
        self.conv1 = nn.Conv2d(1, 64, (5, 5), (1, 1), (2, 2))
        self.modules = self.creat_network()
        self.prePrp = preprocess()

    def creat_network(self):
        models = nn.ModuleList()
        models.append(self.prePrp)
        model = nn.Sequential()
        model.add_module('conv{0}'.format(1),
                         nn.Conv2d(3, 64, 3, 1, 0, bias=False))
        model.add_module('bn{0}'.format(1), nn.BatchNorm2d(3))
        models.append(model)

    def forward(self, x):
        for m in self.modules:
            x = m(x)
        return x

torch_model = SuperResolutionNet()
print(torch_model)

</code></pre>
<p>Thank you very much</p>

ptrblck: <p><code>self.creat_network()</code> tries to access <code>self.prePrp</code> before it’s created, so you would have to move its creation before calling self.creat_network()`:</p>
<pre><code class="lang-python">class SuperResolutionNet(nn.Module):
    def __init__(self):
        super(SuperResolutionNet, self).__init__()
        self.relu = nn.ReLU(inplace=False)
        self.conv1 = nn.Conv2d(1, 64, (5, 5), (1, 1), (2, 2))
        self.prePrp = preprocess()
        self.modules = self.creat_network()
    ...
</code></pre>

yangyi_hpu: <p>Thank you ptrblck. I get it</p>
How to implement back propagation of multiple models that share a portion of their weights

Graviton: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/b/9bbaaa4f7bb2831e48d48fb46e37f7e66ea9163e.png" data-download-href="https://discuss.pytorch.org/uploads/default/9bbaaa4f7bb2831e48d48fb46e37f7e66ea9163e" title="截屏2021-01-24 15.34.25"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/b/9bbaaa4f7bb2831e48d48fb46e37f7e66ea9163e_2_608x500.png" alt="截屏2021-01-24 15.34.25" data-base62-sha1="mdDW3CTFP6V4qilKfqpk4bqxmns" width="608" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/b/9bbaaa4f7bb2831e48d48fb46e37f7e66ea9163e_2_608x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/b/9bbaaa4f7bb2831e48d48fb46e37f7e66ea9163e_2_912x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/b/9bbaaa4f7bb2831e48d48fb46e37f7e66ea9163e_2_1216x1000.png 2x" data-dominant-color="F4F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">截屏2021-01-24 15.34.25</span><span class="informations">1220×1002 52.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Acoording to the above firgure,how to implement to back propagation? The model include two AutoEncoders shared one encoder.How to optim AE1 and AE2 according $L_{AE1}$ and $L_{AE2}$.<br>
I get a solution,but I feel that it was wong.</p>
<pre data-code-wrap="python"><code class="lang-python">        Z = self.encoder(input)
        W1_prime = self.decoder1(Z)
        W2_prime = self.decoder2(Z)
        W2_double_prime = self.decoder2(self.encoder(W1_prime))

        error1 = self.loss1(input, W1_prime, W2_double_prime, i+1)
        error2 = self.loss2(input, W2_prime, W2_double_prime, i+1)

        error1.backward(retain_graph=True)
        error2.backward()
        self.optimizer1.step() # self.optimizer1 = optim.Adam([{'params': self.encoder.parameters()}, {'params': self.decoder1.parameters()}])
        self.optimizer2.step() # self.optimizer2 = optim.Adam([{'params': self.encoder.parameters()}, {'params': self.decoder2.parameters()}])
        self.optimizer1.zero_grad()
        self.optimizer2.zero_grad()
</code></pre>
<p>It seem like that Using $L_{AE1}$ and $L_{AE2}$ to optimise the first autoencoder, the two loss functions are similarly used to optimise the second autoencoder, rather than being independently optimised separately</p>

CedricLy: <p>Use backward() function on total_Loss.<br>
Total_Loss can be the sum of the two losses for example.</p>

Graviton: <p>I cannot use total loss.the second term of two loss functions are opposite numbers.In fact ,the author<br>
uses the idea of GAN</p>
Appending pytorch tensors to a list

yhpknu: <p>I am trying to get a history of values of x with ‘for’ statement.<br>
For python floats, I get the expected list.<br>
The result is surprising with tensors: all entries of the list are the same!<br>
Please let me know why this happens with tensors. Thanks.</p>
<p>— code —</p>
<pre>import torch

# when x is a python float

x_hist = []
x = 1.1

for i in range(3):
    x -= 0.1
    x_hist.append(x)

print(x_hist)  
# out [1.0, 0.9, 0.8]

# when x is a tensor

x_hist = []
x = torch.tensor(1.1)

for i in range(3):
    x -= 0.1
    x_hist.append(x)

print(x_hist)  
# out [tensor(0.8000), tensor(0.8000), tensor(0.8000)]

</pre>

InnovArul: <p>In the case of tensor, <code>x</code> is a reference to the tensor. When we append it to the list, the reference is appended (not the value).<br>
Hence, when you modify <code>x</code> later in-place (<code>x -= 0.1</code>) and print the elements in the list later, they all print the same (latest) value of <code>x</code>.</p>

yhpknu: <p>OK, I see. Thanks.<br>
Then the next question follows:.<br>
what should I do to get the list I want, i.e., [1.0, 0.9, 0.8]?</p>

InnovArul: <p>It depends on your usecase.<br>
If you do not want gradient, you can use <code>.item()</code> or you can use <code>.clone()</code>.</p>

yhpknu: <p>Thanks, again!<br>
I tried both .item() and .clone().</p>
<p>When I use  x.item(), I get<br>
[1.0, 0.8999999761581421, 0.7999999523162842]<br>
and when I use .clone(), I get<br>
[tensor(1.), tensor(0.9000), tensor(0.8000)]</p>
<p>Is it interesting?</p>

InnovArul: <aside class="quote no-group" data-username="yhpknu" data-post="5" data-topic="109821">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yhpknu/48/33785_2.png" class="avatar"> yhpknu:</div>
<blockquote>
<p>[tensor(1.), tensor(0.9000), tensor(0.8000)]</p>
</blockquote>
</aside>
<p>When the tensor is printed, it prints the rounded values up to four digits.</p>

yhpknu: <p>Great answers! Thanks.</p>
AttributeError: 'builtin_function_or_method' object has no attribute 'fftn'

kiranchari: <p>Hi, I am using torch version ‘1.7.1+cu110’ but fftn appears to be missing at torch.fft.fftn.</p>
<p>According to <a href="https://pytorch.org/docs/1.7.1/fft.html#torch.fft.fftn" class="inline-onebox" rel="noopener nofollow ugc">torch.fft — PyTorch 1.7.0 documentation</a> this should exist.</p>
<p>Any advice? Thanks</p>

ptrblck: <p>I guess you might be missing the <code>import torch.fft</code> statement and thus <code>torch.fft.fftn</code> will raise an error.</p>
<pre><code class="lang-python">&gt;&gt;&gt; import torch
tor&gt;&gt;&gt; torch.__version__
'1.7.1'
&gt;&gt;&gt; torch.fft.fftn
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
AttributeError: 'builtin_function_or_method' object has no attribute 'fftn'

&gt;&gt;&gt; import torch.fft
&gt;&gt;&gt; torch.fft.fftn
&lt;built-in function fft_fftn&gt;
</code></pre>

kiranchari: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>, that fixed it. I didn’t see this line in the docs:</p>
<blockquote>
<p>“To use these functions the torch.fft module must be imported since its name conflicts with the <a href="https://pytorch.org/docs/stable/generated/torch.fft.html#torch.fft" rel="noopener nofollow ugc"><code>torch.fft()</code></a> function”</p>
</blockquote>
How do you adjust the bounding box during image augmentation?

Superklez: <p>When image augmentation is applied to input images, how do you adjust the labels of the bounding box? I’m trying to implement data augmentation to a YOLO model.</p>

ptrblck: <p>You could try to reuse some transformations from <a href="https://github.com/pytorch/vision/blob/7dfd8dc8d6e8a5cd2a51d7146c15a7c79729fd95/references/detection/transforms.py"><code>detection/transforms.py</code></a> or use an approach from e.g. <a href="https://discuss.pytorch.org/t/apply-transformations-on-a-set-of-points-boundary-box/27540">this topic</a>.</p>

Eta_C: <p><a href="https://github.com/aleju/imgaug" rel="noopener nofollow ugc">imgaug</a> or <a href="https://github.com/albumentations-team/albumentations" rel="noopener nofollow ugc">albumentations</a> may help you.</p>
<p>Some Examples:<br>
<a href="https://imgaug.readthedocs.io/en/latest/source/examples_bounding_boxes.html" rel="noopener nofollow ugc">imgaug: Examples: Bounding Boxes</a><br>
<a href="https://albumentations.ai/docs/getting_started/bounding_boxes_augmentation/" rel="noopener nofollow ugc">albumentations: Bounding boxes augmentation</a></p>
Would Pytorch move from Github to Opensource/libre code hosting repo?

mathematics: <p>Is it possible to move Pytorch code bases to <code>opensource code hosting repository</code> instead of using proprietary ones.<br>
What do you think of it Pytorch, I need your answer…?<br>
Any plan?<br>
What do you think of it?</p>

ptrblck: <p>What do you think would be the benefit of moving the complete repository?</p>

mathematics: <p>As opensource pytorch moving to open source hosting repo.<br>
We would be at least more closer to freedom than using any proprietary platform that is <a href="https://sanctum.geek.nz/why-not-github.html" rel="noopener nofollow ugc">also bounded by extreme laws</a>. Except freedom feature is more or less good than Github.</p>
<p>I’ve seen big free/opensource projects like debian  like projects are both available to gitlab and github too. I am thinking if our Pytorch would make that decision too.</p>
<p>I was recently got to know FSF and its philosophies, <a href="https://www.gnu.org/software/repo-criteria-evaluation.html" class="inline-onebox" rel="noopener nofollow ugc">GNU Ethical Repository Criteria Evaluations - GNU Project - Free Software Foundation</a> .   Although gitlab is not too pure opensource  but except gitlab with FOOS philosophies are less or not yet developed. (I do not work with any of these)</p>
<p>What do you think of it? Would it be possible?</p>
<p>edit:<br>
I hope you would understand what I am trying to say.<br>
<a href="https://github.com/1995parham/github-do-not-ban-us/" rel="noopener nofollow ugc"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/e/9e6035dfd9c3620655dd1bde1cdee2d239ff8f43.jpeg" alt="image" data-base62-sha1="mB3zm5BAEuBdCnVqJRKYw7EEPJN" width="690" height="66"> </a></p>
<p>I’ve found Gitlab and Github too does that.</p>
<p><em><strong>Github is more about  Politics than Opensource/freedom.</strong></em></p>
Where is the input value for backpropagation saved?

kolo_kolo: <p>Hello dear Pytorch-User,<br>
i have the following question:</p>
<p>suppose you have a very simple network with:</p>
<p><em>input</em>  * w1 = output<br>
MSE(output,target) = loss</p>
<p>Then you need the value from <em>input</em> for the backpropagation. But i can t find the place where it is saved. I allready read a little bit about the backpropagation system from pytorch, but it didnt give me the answer to this question. I am just curious, its not very important.</p>

albanD: <p>Hi,</p>
<p>It is saved within the computational graph in c++ (it saves a reference to input).<br>
There are no easy way for you to access that reference from python though.</p>

kolo_kolo: <p>hi, thank you very much for your answer</p>
Convolution Implementation

zahirbilal: <p>Hello,</p>
<p>I am trying to implement the convolution (or to be accurate the cross-correlation) function  with a certain padding and stride value</p>
<pre><code class="lang-auto">x = torch.randn(1, 3, 6, 8) # input
kernel = torch.randn(2, 3, 4, 5) # filter

conv = Conv2d(kernel, padding=1, stride=3)
</code></pre>
<p>I have already implemented the padding function,<br>
my Q is how can I implement the Conv2d function in python/pytorch to calculate the cross correlation?<br>
I tried with nested looping and summation but that was inefficient. a seperate variable for stride (self.stride) should be used.</p>
<p>P.S: the whole intuition of the task is to not use the pre-implemented Conv2d</p>
<p>Thanks,</p>

ptrblck: <aside class="quote no-group" data-username="zahirbilal" data-post="1" data-topic="108769">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zahirbilal/48/33374_2.png" class="avatar"> zahirbilal:</div>
<blockquote>
<p>I tried with nested looping and summation but that was inefficient.</p>
</blockquote>
</aside>
<p>You could try to use <code>unfold</code> to create the input patches and a <code>matmul</code> to implement the convolution as given in the <a href="https://pytorch.org/docs/stable/generated/torch.nn.Unfold.html#torch.nn.Unfold">Unfold docs</a>.<br>
Note that this would most likely still be slower than the native implementations.</p>
How to get output of a model by inserting feature vector to a specific layer?

miker2808: <p>Hello, I couldn’t find an answer.<br>
Considering I have a model like resnet, and I would like to receive the output. But, instead of inserting a feature vector to the begining, I would like to insert the feature vector to a specific layer of the model. by this, I’m avoiding few starting layers of the network.<br>
is it possible? without creating a new model?</p>
<p>Thanks in advance!</p>

ptrblck: <p>Depending on the architecture of the model you could either pass the new input to the desired modules directly:</p>
<pre><code class="lang-python">model = ...
input = ...
output = model.last_block(input)
</code></pre>
<p>In case the architecture is not split into submodules and uses e.g. the functional API in its <code>forward</code>, I would recommend to write a new custom <code>nn.Module</code> and adapt the <code>forward</code> method as needed.</p>
Run forward from specific layer

Serfati: <p>Hi Everyone,<br>
I’m using a pretrained Resnet18 model, my main challenge is to run a feature vector that i already have<br>
(taken from another model last <code>avgpool</code> layer) and finish the run with the help of the last <code>fc</code> layer and detect all objects that this feature vector represent.</p>
<p>some code</p>
<pre><code class="lang-python">np_fv = np.load('feature_vector.npy')
fv = torch.from_numpy(np_fv)
fv.shape #torch.Size([512])

# Load the pretrained model
model = models.resnet18(pretrained=True)

# Use the model object to select the desired layer
layer = model._modules.get('fc')

# Set model to evaluation mode
model.eval()

#imageNet has 1k classes so fc output dim is 1k
my_embedding = torch.zeros(1000)
def copy_data(m, i, o):
        my_embedding.copy_(o.data)
h = layer.register_forward_hook(copy_data)

model.avgpool =  # what should I do next?
</code></pre>
<p>how do i get prediction only from <strong>feature_vector</strong> without the input image?</p>

ptrblck: <p>If you only want to use the <code>fc</code> layer of the pretrained resnet, you could use it as:</p>
<pre><code class="lang-python">fv = ...
output = model.fc(fv)
</code></pre>
<p>I’m not sure, if I misunderstand the question or why you need forward hooks.</p>
Any workaround for loading a newer version model weight with older verison pytorch

Alpha: <p>I am using pytorch1.6 and can’t use the latest 1.7 for some reason.</p>
<p>When I load the model, I encounter the error:</p>
<pre><code class="lang-auto">    super(_open_zipfile_reader, self).__init__(torch._C.PyTorchFileReader(name_or_buffer))
RuntimeError: version_ &lt;= kMaxSupportedFileFormatVersion INTERNAL ASSERT FAILED at ../caffe2/serialize/inline_container.cc:132, please report a bug to PyTorch. Attempted to read a PyTorch file with version 4, but the maximum supported version for reading is 3. Your PyTorch installation may be too old.
</code></pre>
<p>I am wondering is there a workaround to solve this problem.</p>
<p>For example, load model with pytorch1.7, and save the weight in a format that pytorch1.6 can use.<br>
The use pytorch1.6 to reload the model.</p>
<p>Is this possible to realize?</p>
<p>Thanks in advance.</p>

Alpha: <p>Following <a href="https://discuss.pytorch.org/t/attempted-to-read-a-pytorch-file-with-version-4-but-the-maximum-supported-version-for-reading-is-3-your-pytorch-installation-may-be-too-old/85412/17"> DableUTeeF</a>,</p>
<p>In my case, I opened the weight file using <code>vim</code> and noticed that at the end of the first line, there is something like this:</p>
<pre><code class="lang-auto">ZZZZZZZZZZZZZZZ4
</code></pre>
<p>I just change <code>4</code> to <code>3</code> and now I can load the model <img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=9" title=":thinking:" class="emoji" alt=":thinking:"> <img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=9" title=":thinking:" class="emoji" alt=":thinking:"></p>
<p>I don’t know if it is OK.</p>

ptrblck: <p>PyTorch doesn’t guarantee forward compatibility, so while your “hack” might work, you might encounter unwanted side effects and issues.<br>
If possible, try to save and load the model in <code>1.6</code> or update to <code>1.7.1</code> (which would be the better approach).</p>
Dataloader for multiple datasets using different batch sizes

cmplx96: <p>Hi all,</p>
<p>I have two datasets of images.</p>
<p>I combined them using <code>torch.utils.data.ConcatDataset</code> and passed the combined dataset to <code>torch.utils.data.DataLoader</code>.</p>
<p>However, now I can only select one batch size. Is there a way to set individual batch sizes to both datasets?</p>
<p>Thanks!</p>

ptrblck: <p>No, that’s not possible using <code>ConcatDataset</code> since this new dataset is treated as one.<br>
If you need different batch sizes, you could create two <code>DataLoader</code>s and set the desired batch size accordingly.</p>
Can I use different learning rate values?

jiho: <p>I have four types of input images, and I want to know that can I set different learning rates for each types?</p>
<p>If it is possible, let me know how to implement it and some keywords about it</p>

ptrblck: <p>You could use different optimizers for each input type and call it w.r.t to the currently used input.<br>
The use case won’t work if you are working with a batch of images containing multiple types of inputs and you would have to define a “default” optimizer.</p>
Modifying stem and use pretrained weights

jahan123: <p>If I modify the <code>stem()</code> for torchvision models, will I be able to use the pretrained wieghts?</p>
<p>I am changing the input layer channels:</p>
<pre><code class="lang-auto">class modifybasicstem(nn.Sequential):
    """The default conv-batchnorm-relu stem
    """
    def __init__(self):
        super(modifybasicstem, self).__init__(
            nn.Conv3d(1, 64, kernel_size=(3, 3, 3), stride=(1, 2, 2),
                      padding=(1, 3, 3), bias=False),
            nn.BatchNorm3d(64),
            nn.ReLU(inplace=True))
#
model = torchvision.models.video.mc3_18(pretrained=False)
model.fc = nn.Sequential(
    nn.Linear(model.fc.in_features, num_classes)
)

model = torchvision.models.video.r3d_18(pretrained=True)
model.stem = modifybasicstem()
model.to('cuda:0')

</code></pre>
<p>Thankyou.</p>

ptrblck: <p>Yes, your code would replace the <code>.stem</code> module only while all other layers would still use their pretrained parameters.</p>
Weights not getting updated when creating tensor in the middle of the computation

finewith_me: <p>I want W to get updated, and as you can see, “x” variable acts as an index variable, which helps in getting relevant weights from W and then combine them to get predictions. But the way I have done it does not seem to update “W”. Also note that if I make requries_grad=False where it is True now in the following code, I get “tensor 0 does not require grad” kinda error. I will be grateful if I can get some explanation why this does not work rather than the workarounds</p>
<pre><code class="lang-auto">x=[torch.tensor([0,1]),torch.tensor([[2,3],[0,1]]),torch.tensor([4,5])]
W=torch.tensor([0.1,0.2,0.3,0.4,0.5,0.6])
# W[x[0]]
optimizer = torch.optim.Adam([W], lr=0.001) #, weight_decay=5e-4)
y=torch.tensor([0.,1.,1.])
for epoch in range(3):
#     W_rel=[W[i] for i in x]
    pred=torch.tensor([torch.sigmoid(torch.sum(W[i])) for i in x],requires_grad=True)
    loss=torch.nn.BCELoss()(pred,y)
    loss.backward()
    optimizer.step()
    print(W)
</code></pre>

ptrblck: <p>You would have to initialize <code>W</code> as an <code>nn.Parameter</code> or set it <code>requires_grad</code> attribute to <code>True</code> in the creation.<br>
Also don’t wrap outputs in new tensors, as this will detach the computation graph.<br>
This code snippet should work:</p>
<pre><code class="lang-python">x=[torch.tensor([0,1]),torch.tensor([[2,3],[0,1]]),torch.tensor([4,5])]
W=torch.tensor([0.1,0.2,0.3,0.4,0.5,0.6], requires_grad=True)
optimizer = torch.optim.Adam([W], lr=0.001) #, weight_decay=5e-4)
y=torch.tensor([0.,1.,1.])
for epoch in range(3):
    pred= torch.stack([torch.sigmoid(torch.sum(W[i])) for i in x])
    loss=torch.nn.BCELoss()(pred,y)
    loss.backward()
    optimizer.step()
    print(W)
</code></pre>
Network DenseNet

randinoo: <p>HI, can i help me to create the model DenseNets?  plz  <img src="https://discuss.pytorch.org/images/emoji/apple/pensive.png?v=9" title=":pensive:" class="emoji" alt=":pensive:"></p>

ptrblck: <p><code>torchvision</code> provides implementations of DenseNets as seen <a href="https://github.com/pytorch/vision/blob/7dfd8dc8d6e8a5cd2a51d7146c15a7c79729fd95/torchvision/models/densenet.py">here</a>.</p>

randinoo: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>  thank you, but it doesn’t work <img src="https://discuss.pytorch.org/images/emoji/apple/pensive.png?v=9" title=":pensive:" class="emoji" alt=":pensive:"> <img src="https://discuss.pytorch.org/images/emoji/apple/pensive.png?v=9" title=":pensive:" class="emoji" alt=":pensive:"> <img src="https://discuss.pytorch.org/images/emoji/apple/pensive.png?v=9" title=":pensive:" class="emoji" alt=":pensive:"></p>
<pre><code class="lang-auto">/usr/local/lib/python3.6/dist-packages/torch/_jit_internal.py in _overload_method(func)
    610         existing_lineno = _overloaded_method_class_fileno[(qual_name, class_name)]
    611         if existing_lineno != line_no:
--&gt; 612             raise RuntimeError("Cannot currently overload the same method name in two different"
    613                                " classes with the same name in the same module")
    614 

RuntimeError: Cannot currently overload the same method name in two different classes with the same name in the same module
</code></pre>
Load_state_dict V/S copy.deepcopy

chinmoy_samant: <p>I have 3 separate datasets lets say D1, D2, D3.</p>
<p>I want to train the same model on the 3 datasets. But as <span class="hashtag">#samples</span>(D1)&gt;#samples(D2)&gt;#samples(D3),<br>
I want to train a model M1 on D1. After the training is complete I want to use these trained weights to initialize model M2 and train D2, and similarly for M3 and D3.</p>
<p>I am currently doing it like this</p>
<p>Make model 1<br>
Train model 1<br>
torch.save(model1.state_dict(), path1)</p>
<p>Make model 2<br>
model2.load_state_dict(torch.load(path1))<br>
Train model 2<br>
torch.save(model2.state_dict(), path2)</p>
<p>Make model 3<br>
model3.load_state_dict(torch.load(path2))<br>
Train model 3<br>
torch.save(model3.state_dict(), path3)</p>
<p>But this is degrading my results for Model 1 and Model 2. Does model2.load_state_dict(torch.load(path1)) change model1’s parameters? If yes, how can I solve my problem?</p>
<p>Thanks</p>

Jesus_Cevallos: <p>The answer should be no, load_state_dict operation doesn’t alters model1’s params. Look: <a href="https://discuss.pytorch.org/t/deepcopy-vs-load-state-dict/61527" class="inline-onebox">Deepcopy vs. load_state_dict</a></p>
What does .view(-1) do?

Geoffrey_Payne: <p>I see this line of code</p>
<pre><code class="lang-auto">lab=lab.view(-1)
</code></pre>
<p>I am wondering what this code does? I put a breakpoint on it, and as far as I can see the contents of lab are the same both before and after I execute the code. So does this code make any difference?</p>

ptrblck: <p>The <code>view(-1)</code> operation flattens the tensor, if it wasn’t already flattened as seen here:</p>
<pre><code class="lang-python">x = torch.randn(2, 3, 4)
print(x.shape)
&gt; torch.Size([2, 3, 4])
x = x.view(-1)
print(x.shape)
&gt; torch.Size([24])
</code></pre>
<p>It’ll modify the tensor metadata and will not create a copy of it.</p>
Problem about pytorch1.1 to pytorch1.7

kuaidiyuan: <p>I successfully run forward in pytorch1.7, and the weight trained in pytorch1.7 appeared noise in the images after run forward.The weight trained on pytorch1.1 is no problem after run forward with pytorch1.7. Do you have any suggestions for me to find the problem? Thank you very much.</p>

ptrblck: <p>Could you explain your issue a bit more, please?<br>
Are you able to train a model in a specific version, but the inference in another PyTorch release changes?</p>
Loss and output(model)

Celmar: <p>The last layer of my model is <code>self.linear = nn.Linear(4096, 2)</code>. Usually to find out which class <code>data</code> belongs to I write</p>
<pre><code class="lang-auto">output = model(data) #which return a tensor of size 2
pred = output.data.max(1, keepdim=True)[1]
</code></pre>
<p>Only I want the probability that <code>data</code> belongs to the first or second class, so I thought I would write :</p>
<pre><code class="lang-auto">output = model(data) #which return a tensor of size 2
pred = torch.nn.Softmax(output) # pred = tensor([Prob(y=0|data), Prob(y=1|data)]) ?
</code></pre>
<p>But i does not work it gives me tensor of size two with negatives values. Do you have an idea to get this probabilities ?</p>
<p>Thanks</p>

ptrblck: <p><code>nn.Softmax</code> is an <code>nn.Module</code> so you would either have to create the module first and call it with <code>output</code> or use <code>F.softmax</code> instead:</p>
<pre><code class="lang-python">pred = torch.nn.Softmax(dim=1)(output)
# or
pred = F.softmax(output, dim=1)
</code></pre>
How to Sample Datasets for Training, Validation and Testing

moreshud: <p>I have collections of CT scan medical image datasets and I would like to sample the dataset for training, validation, and testing due to the computational challenge of not being able to use all the datasets. A large number of the images are with/without pathologies - GGO and CON while others only have either GGO or CON.</p>
<p>As I need to sample/split the whole datasets into training, validation, and testing, what approach(es) do you suggest in order to have an approximate statistic distribution of the validation/testing in training?</p>

moreshud: <p><a class="mention" href="/u/kfrank">@KFrank</a> <a class="mention" href="/u/ptrblck">@ptrblck</a> Your contributions will be highly appreciated.</p>

MrPositron: <p>Check the random split function: <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.random_split" class="inline-onebox" rel="noopener nofollow ugc">torch.utils.data — PyTorch 1.7.0 documentation</a></p>
RuntimeError: value cannot be converted to type at::Half without overflow: -1e+30

Seo: <p>Hi,<br>
I am implementing my training with mixed-precision operations with <code>torch.cuda.amp.autocast(enabled=True)</code> and I have some trouble with a masking operation (for the transformer model). What I need to do is to mask all the 0’s before the softmax computation. In order to do so, I use a very negative value for the mask (that ensures the softmax to output 0 attention for those logits). Unfortunately, with mixed precision this operation causes overflow. The operation is the following:</p>
<pre><code class="lang-auto">_MASKING_VALUE=-1e30
masked_attn_logits = attn_logits.masked_fill(attn_mask==0, value=_MASKING_VALUE)
</code></pre>
<p>I already tried to reduce the masking value to -1e6 but without success. Can you help me?</p>

Seo: <p>solution:<br>
<code>_MASKING_VALUE = -1e+30 if attn_logits.dtype == torch.float32 else -1e+4</code></p>