akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Usage of detach()

oasjd7: <pre><code class="lang-auto">total_loss += loss(A, B.detach())
total_loss += loss(B, A.detach())
total_loss.backward()
</code></pre>
<p>Hi, all</p>
<p>If I call <code>B.detach()</code> in the front, B cannot be trained after this code?</p>

albanD: <p>Hi,</p>
<p>What do you mean by “in the front” ?</p>
Torch.nn.functional.conv2d on 5D vectors?

adamr: <p>I understand that the intended usage of the <code>conv2d</code> is to make it act on 4D input vectors and kernels, the docs never specifically stated that you can’t use it with higher dimensions.</p>
<p>I tried, for fun, testing what does <code>conv2d</code> does for 5D vectors. First I tried to try the simplest no-op case, that should just return multi-dimensional vector of size 1:</p>
<pre><code class="lang-python">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.nn.functional.conv2d(torch.ones(1,1,1,1), torch.ones(1,1,1,1))
tensor([[[[1.]]]])
&gt;&gt;&gt; torch.nn.functional.conv2d(torch.ones(1,1,1,1,1), torch.ones(1,1,1,1,1))
RuntimeError: expected stride to be a single integer value or a list of 3 values to match the convolution dimensions, but got stride=[1, 1]
</code></pre>
<p>I’ve got the error message asking me to put the correct (three) dimensions on the stride, padding, dilation and… output_padding.</p>
<pre><code class="lang-python">&gt;&gt;&gt; torch.nn.functional.conv2d(torch.ones(1,1,1,1,1), torch.ones(1,1,1,1,1), stride=(1,1,1), padding=(0,0,0), dilation=(1,1,1), output_padding=(0,0,0))
TypeError: conv2d() got an unexpected keyword argument 'output_padding'
</code></pre>
<p>Is it the pytorch bug? Or did I miss the limitation that the conv2d can only act on 4D vectors - if so, why the code does not check for it <code>assert len(input.shape)==4</code> when it already checks for so many errors?</p>

albanD: <p>Hi,</p>
<p>Yes, conv2d can only work with 4D inputs. You can use conv3d to work with 5D.<br>
And we are definitely missing some error checking here to make sure the user is not providing wrong arguments. Could you open an issue on github about that please?</p>
Is it mandatory to have batch size power of 2 on gpu?

hadaev8: <p>Does pytorch or cuda have any specific optimization or something?</p>

albanD: <p>Hi,</p>
<p>No it is not mandatory.<br>
And power of 2 are not particularly important either.<br>
Maybe powers of 32 that are the size of the streaming multiprocessors? But even that depends a lot on how the cuda kernel is implemented and, in general, won’t lead to any significant difference.</p>
Meta learning with Transformers

kl_divergence: <p>I want to train my transformer model with MAML gradient updates. I’ve found that <code>higher</code> is an efficient way to go about it. I’m open to other methods also.</p>
<p>So the <a href="https://github.com/prajjwal1/fluence/blob/master/fluence/meta/meta_dataset.py" rel="nofollow noopener"><code>MetaDataset</code> </a> wraps any <code>GlueDataset</code> to give a list containing all classes when <code>meta_dataset[0]</code> is called. So this will become, <code>num_of_classes (N)</code> way K shot example.</p>
<p>I’ve written this, which extends HF <code>Trainer</code> for MAML.</p>
<pre><code class="lang-auto">def train(self):

        self.create_optimizer_and_scheduler(
            int(
                len(self.train_dataloader)
                // self.args.gradient_accumulation_steps
                * self.args.num_train_epochs
            )
        )

        logger.info("***** Running training *****")

        self.global_step = 0
        self.epoch = 0

        eval_step = [2 ** i for i in range(1, 20)]
        inner_optimizer = torch.optim.SGD(
            self.model.parameters(), lr=self.args.step_size
        )
        self.model.train()

        tqdm_iterator = tqdm(self.train_dataloader, desc="Batch Index")

        #  n_inner_iter = 5
        self.optimizer.zero_grad()
        query_dataloader = iter(self.train_dataloader)

        for batch_idx, meta_batch in enumerate(tqdm_iterator):
            target_batch = next(query_dataloader)
            outer_loss = 0.0
            # Loop through all classes
            for inputs, target_inputs in zip(meta_batch, target_batch):

                for k, v in inputs.items():
                    inputs[k] = v.to(self.args.device)
                    target_inputs[k] = v.to(self.args.device)

                with higher.innerloop_ctx(
                    self.model, inner_optimizer, copy_initial_weights=False
                ) as (fmodel, diffopt):

                    inner_loss = fmodel(**inputs)[0]
                    diffopt.step(inner_loss)
                    outer_loss += fmodel(**target_inputs)[0]

            self.global_step += 1
            self.optimizer.step()

            outer_loss.backward()

            if (batch_idx + 1) % self.args.gradient_accumulation_steps == 0:
                torch.nn.utils.clip_grad_norm_(
                    self.model.parameters(), self.args.max_grad_norm
                )

            # Run evaluation on task list
            if self.global_step in eval_step:
                output = self.prediction_loop(self.eval_dataloader, description = "Evaluation")
                self.log(output.metrics)

                output_dir = os.path.join(
                    self.args.output_dir, f"{PREFIX_CHECKPOINT_DIR}-{self.global_step}",
                )
                self.save_model(output_dir)
</code></pre>
<p>The above code doesn’t seem to work properly as in the accuracy doesn’t improve. Any directions/tips would be appreciated.</p>
Runaway gradient?

Topsoil: <p>I am playing around with autoencoders, and my network has this structure:<br>
Encoder:</p>
<pre><code class="lang-auto">self.main = nn.Sequential(
        nn.Conv2d(3,500,3,1,padding = 1),
        nn.ReLU(),
        nn.Conv2d(500,450,3,1,padding = 1),
        nn.ReLU(),
        nn.AvgPool2d(2,2,0),
        nn.Conv2d(450,400,3,1,padding = 1),
        nn.ReLU(),
        nn.AvgPool2d(2,2,0),
        nn.Conv2d(400,350,3,1,padding = 1),
        nn.ReLU(),
        nn.AvgPool2d(2,2,0),
        nn.Conv2d(350,300,3,1,padding = 1),
        nn.ReLU(),
        nn.AvgPool2d(2,2,0),
        nn.Conv2d(300,250,3,1,padding = 1),
        nn.ReLU(),
        nn.AvgPool2d(2,2,0),
        nn.Conv2d(250,200,3,1,padding = 1),
        nn.ReLU(),
        nn.AvgPool2d(2,2,0),
        nn.Conv2d(200,150,3,1,padding = 1),
        nn.ReLU(),
        nn.AvgPool2d(2,2,0),
        nn.Conv2d(150,125,3,1,padding = 1),
        nn.ReLU(),
        nn.AvgPool2d(2,2,0),
        nn.Conv2d(125,100,3,1,padding = 1),
        nn.ReLU(),
    )
</code></pre>
<p>Decoder:</p>
<pre><code class="lang-auto">self.main = nn.Sequential(
        nn.Upsample(scale_factor = 2),
        nn.Conv2d(100,125,3,1,padding = 1),
        nn.ReLU(),
        nn.Upsample(scale_factor = 2),
        nn.Conv2d(125,150,3,1,padding = 1),
        nn.ReLU(),
        nn.Upsample(scale_factor = 2),
        nn.Conv2d(150,200,3,1,padding = 1),
        nn.ReLU(),
        nn.Upsample(scale_factor = 2),
        nn.Conv2d(200,250,3,1,padding = 1),
        nn.ReLU(),
        nn.Upsample(scale_factor = 2),
        nn.Conv2d(250,300,3,1,padding = 1),
        nn.ReLU(),
        nn.Upsample(scale_factor = 2),
        nn.Conv2d(300,350,3,1,padding = 1),
        nn.ReLU(),
        nn.Upsample(scale_factor = 2),
        nn.Conv2d(350,400,3,1,padding = 1),
        nn.ReLU(),
        nn.Upsample(scale_factor = 2),
        nn.Conv2d(400,450,3,1,padding = 1),
        nn.ReLU(),
        nn.Conv2d(450,500,3,1,padding = 1),
        nn.ReLU(),
        nn.Conv2d(500,3,3,1,padding = 1)
    )
</code></pre>
<p>but i’ve noticed that for some reason after some training time loss skyrockets from 0.0X to hundreds and beyond. What might be the culprit?</p>
<p>Optimizer settings are these:</p>
<pre><code class="lang-auto">optimizer = optim.Adam(model.parameters(),lr = 0.002, betas=(0.5,0.999));
</code></pre>

AlphaBetaGamma96: <p>Have you tried playing around with the hyperparameters of the Optimizer or perhaps used another Optimizer like SGD?</p>
RuntimeError: Given input size: (512x1x1). Calculated output size: (512x0x0). Output size is too small

Archy_dragon: <p>can anyone help i develop cnn and this error pop up and i have no idea what went wrog or how to fix it can annyone help</p>
<pre><code class="lang-auto">class gface1(nn.Module):
    def __init__(self, bottleneck_setting=MobiFace_bottleneck_setting, final=False):
        super(gface1, self).__init__()
        self.final = final

        self.conv1 = ConvBlock(3, 64, 3, 2, 1)

        self.dw_conv1 = ConvBlock(64, 64, 3, 1, 1, dw=True)

      
        self.conv4= ConvBlock6(64,128,3, 1, 1)

        self.conv5= ConvBlock6(128,256,3, 1, 1)

        self.conv6= ConvBlock6(256,512,3, 1, 1)

        #self.conv7 = ConvBlock(512, 512, 1, 1, 0, linear=True)

        #self.linear1 = nn.Linear(512*7*7, 512)

        #self.linear7 = ConvBlock4(512, 512, (7, 6), 1, 0, dw=True, linear=True)

        #self.linear1 = ConvBlock4(512, 128, 1, 1, 0, linear=True)

        #self.prelu1 = nn.PReLU()



    def forward(self, x):
        x = self.conv1(x)
        x = self.dw_conv1(x)
        #x= self.conv3(x)
        x=self.conv4(x)
        x= self.conv5(x)
        x=self.conv6(x)
        #x=self.conv7(x)
        #x = x.view(x.size(0), -1)
        #x = self.linear1(x)
        #if self.final is False:
         # x = self.prelu1(x)
        #x = self.linear7(x)
        #x = self.linear1(x)
        #x = x.view(x.size(0), -1)
        return x
        
        


</code></pre>
<pre><code class="lang-auto">class ConvBlock6(nn.Module):
    def __init__(self, inp, oup, k, s, p, dw=False, linear=False):
        super(ConvBlock6, self).__init__()
        self.linear = linear
        self.conv = nn.Conv2d(inp, oup, k, s, p, groups=inp, bias=False)
        self.batch1 = nn.BatchNorm2d(oup)
        self.PRelu1=nn.PReLU(oup)
        self.max1= nn.MaxPool2d((2, 2), 2)


        self.dw_conv2=nn.Conv2d(oup, oup, k, s, p,groups=inp * 2, bias=False)
        self.batch2=nn.BatchNorm2d(oup)
        self.PRelu2=nn.PReLU(oup)
        self.max2=  nn.MaxPool2d((2, 2), 2)
        

        self.conv3= nn.Conv2d(oup, oup, 1, 1, 0, bias=False)
        self.batch3=nn.BatchNorm2d(oup)


        

        #self.maxP = x

    def forward(self, x):
        x = self.conv(x)
        x = self.batch1(x)
        x=self.PRelu1(x)
        x= self.max1(x)
        x=self.dw_conv2(x)
        x=self.batch2(x)
        x=self.PRelu2(x)
        x=self.max2(x)
        x=self.conv3(x)
        x=self.batch3(x)
        return self.conv3(x)

</code></pre>

Topsoil: <p>at some step in your forward pass your input gets downsampled too much, likely because of one extra pooling layer. during your forward pass report shape of tensor, and find where the problem is that way</p>

Archy_dragon: <p>so remove or reduce pool layer should help am i correct</p>

Topsoil: <p>Or change its settings, or the proble might be in convolutions, they also can change the size of channels</p>
Vectorize for loop in variable length of indexes

zzhu: <p>Hi, I would like to know if there is any better way to vectorize following calculation instead of for loops:</p>
<p>I have a 2d matrix A and an index matrix B (only 0 and 1) of the same shape. For each column of A, I want to take the row index from B where B is 1, then take the corresponding rows from A, then do another for loop of an iterative sum:<br>
<code>for i in row_index:     A[1: i] = A[1: i] * w1 + A[0: i-1] * w2</code></p>
<p>And the key point is that the index in each column of B might be of different lengths.</p>
<p>I would like to know if there is any way to better vectorize above instead of doing two for loops. Any idea is appreciated, thanks!</p>

googlebot: <p>this looks recurrent,e.g. for scalar cells:</p>
<pre><code class="lang-auto">a1_new=a1*w1+a0*w2
a2=a2*w1+a1_new*w2
</code></pre>
<p>if I got the computation correctly, it is not parallelizable.</p>
<p>However outer loop should be vectorizable, as columns are independent. Basic idea is to convert indexes into a mask, in your case this means expanding w1 and w2 into 2d arrays, and masking by w1=1,w2=0. After that you should be able to process all columns at once.</p>
Pytorch example word embedding takes a very long time for each epoch?

HenrikManolia: <p>I’m using the example N gram language modelling on my own dataset, which after creating the trigrams, i have 4 million trigrams.</p>
<p>the problem I’m having is its extremely slow, for example just  1 epoch over the 4 million trigram takes 48 hours to complete! is this normal? after moving it to GPU it takes 20 minutes which still is a lot, so is this normal?</p>
<p>I have a NVMe ssd with i7 6700k with 16gb of ram and Nvidia 980 Ti so its not a slow system either</p>
How to use Cosine Annealing?

gamebo5000: <p>I am still new to PyTorch and I am going off this link: <a href="https://pytorch.org/docs/stable/optim.html" rel="nofollow noopener">https://pytorch.org/docs/stable/optim.html</a> I don’t see many examples of it being applied online so this is how I thought it should look.</p>
<pre><code class="lang-auto"> Q = math.floor(len(train_data)/batch)

 lrs = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max = Q)
</code></pre>
<p>Then in my training loop, I have it set up like so:</p>
<pre><code class="lang-auto"> # Update parameters
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        lrs.step()
</code></pre>
<p>For the training loop, I even tried a different approach such as:</p>
<pre><code class="lang-auto"> # Update parameters
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        

        # Print interim results
        if b%10 == 0:
            print(f'epoch: {epoch:2}  batch: {b:4} loss: {loss.item():10.8f}  \
accuracy: {accuracy:3.3f}%')
            
        if b%214 == 0:

            train_acc.append(accuracy)
            train_losses.append(loss)
            train_correct.append(trn_corr)
            mean_loss = sum(train_losses)/len(train_losses)
            
            for param_group in optimizer.param_groups:
                lr_track.append(param_group['lr'])
            print("\n")
            lrs.step(mean_loss)
</code></pre>
<p>Where after every epoch it gets adjusted.</p>

Worulz: <p>An example of implement Cosine Annealing + warm restarts can be found here.</p>
<p><a href="https://discuss.pytorch.org/t/how-to-implement-torch-optim-lr-scheduler-cosineannealinglr/28797/6">How to implement torch.optim.lr_scheduler.CosineAnnealingLR?</a></p>
<p>hope that helps!</p>
Get main diagonal of two large matrix multiplication

xdwang0726: <p>I am trying to get the main diagonal from the multiplication of two large matrices. Here is my implementation:</p>
<pre><code class="lang-auto">def col_wise_mul(m1, m2):
    result = torch.zeros(0)
    for i in range(m1.shape[1]):
        v1 = m1[:, i, :]
        v2 = m2[:, i]
        v = torch.matmul(v1, v2).unsqueeze(1)
        result = torch.cat((result, v), dim=1)
    return result
</code></pre>
<p>I know that I could multiply two matrices first and then get the diagonal like below:</p>
<pre><code class="lang-auto">x = torch.diagonal(torch.matmul(x_feature, label_feature), offset=0).transpose(0, 1)
</code></pre>
<p>As the two matrices are very large, I am wondering is there any improvements that could done to my implementation (basically I want to avoid loop as this part will happen in the forward)?</p>
<p>Thanks!</p>

albanD: <p>Hi,</p>
<p>I think the best way to do this is to rewrite (assuming 2D matrices) <code>matmul(m1, m2).diag()</code> to <code>(m1 * m2.t()).sum(dim=1)</code>.<br>
You can handle the extra batch dimension if you have one by just changing the the dimension you sum over.</p>

xdwang0726: <p>Thanks! It really helps!</p>
Impact of CNN oupt concat size in ensemble

janderbell: <p>Hello,</p>
<p>I have a model where the output of two CNNs is concatenated to another vector before a final classification output. I am just wondering what impact the dimension of the CNN output has on the final model. I have seen people use size 1 and size 10 before.</p>
<p>Many thanks</p>

ptrblck: <p>I guess you are referring to the “dimension” of the feature tensors from both CNNs before concatenation?<br>
If so, I think the same effects can be observed as in a “standard” model (i.e. without the model ensemble).<br>
I.e. the smaller the feature tensor the more compressed the information is and vice versa. The overall performance would most likely depend on the actual use case.</p>
<p>That being said, note that sometimes the penultimate activations are used from pretrained models, which have a fixed dimension. If you don’t add another layer on this feature extractor, you would just go with the pretrained model architecture.</p>
How to cross-layer parameters sharing?

athor_M: <p>hi, i am trying to study ALBERT, and i have question, how to cross-layer parameters sharing?</p>
<p>my code like this<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/3/f3c8705aa452edea95d396687e50d3d4170e8342.png" data-download-href="https://discuss.pytorch.org/uploads/default/f3c8705aa452edea95d396687e50d3d4170e8342" title="V8`P}TVVEUNVFI4PK_RR"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/3/f3c8705aa452edea95d396687e50d3d4170e8342_2_690x420.png" alt="V8`P}TVVEUNVFI4PK_RR" data-base62-sha1="yMBsKfIUfXO4jQokZ4EWJ8aABJ8" width="690" height="420" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/3/f3c8705aa452edea95d396687e50d3d4170e8342_2_690x420.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/3/f3c8705aa452edea95d396687e50d3d4170e8342_2_1035x630.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/3/f3c8705aa452edea95d396687e50d3d4170e8342.png 2x" data-dominant-color="242524"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">V8`P}TVVEUNVFI4PK_RR</span><span class="informations">1255×765 106 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<pre><code class="lang-auto">class Encoder(nn.Module):
    def __init__(self, hparams):
        super(Encoder, self).__init__()
        self.encoder_layer = EncoderLayer(hparams)

    def forward(self, enc_inputs):
        for i in range(11):
            # enc_outputs: [batch_size, src_len, d_model]
            enc_inputs = self.encoder_layer(enc_inputs, enc_self_attn_mask)
    return enc_inputs
</code></pre>
<p>i am not sure parameters of <code>enc_self_attn</code> and <code>pos_ffn</code> are shared in each loop</p>
<p>thanks for you answer very much</p>
Overlap annotation bar chart

shangguan91: <p>is there any way to solve the annotation of overlap, or increase the size of chart, or the range of X-axis</p>
<pre><code class="lang-auto">import matplotlib
import matplotlib.pyplot as plt
import numpy as np

label_list=['PointNet',
'PointNet++',
'EdgeConv',
'PointCNN',
'Convpoint',
'LightConvPoint',
'SplineCNN',
'RSConv']

y1 = [90,
90,
85,
90,
95,
95,
87.5,
75
]
y2= [89.2,
90.7,
91.7,
92.2,
92.5,
92.5,
92.65,
93.6]

x = np.arange(len(label_list))  # the label locations
width = 0.4 # the width of the bars

fig, ax = plt.subplots()
rects1 = ax.bar(x - width/2, y1, width, label='Church')

rects2 = ax.bar(x + width/2, y2, width, label='Modelnet40')

# Add some text for labels, title and custom x-axis tick labels, etc.
ax.set_ylim(70,100)
ax.set_ylabel('Best achieved Test accuracy%')
ax.set_title('performance comparison modelnet40 and Churchdata')

ax.set_xticks(x)

ax.set_xticklabels(label_list,rotation='vertical')




ax.legend(loc='lower left')    

def autolabel(rects):
    """Attach a text label above each bar in *rects*, displaying its height."""
    for rect in rects:
        height = rect.get_height()
        ax.annotate('{}'.format(height),
                    xy=(rect.get_x() + rect.get_width() / 2, height),
                    xytext=(0, 3),  # 3 points vertical offset
                    textcoords="offset points",
                    ha='center', va='bottom')



autolabel(rects1)
autolabel(rects2)



plt.show()
</code></pre>

ptrblck: <p>This seems to be a <code>matplotlib</code>-specific question so I would recommend to post it on StackOverflow, where you would probably get a faster and better answer. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
3D U-net output interpretation

krioux5: <p>Hello,</p>
<p>I am trying to utilize the following open source 3D Unet: <a href="https://github.com/wolny/pytorch-3dunet" rel="nofollow noopener">https://github.com/wolny/pytorch-3dunet</a></p>
<p>The repository includes links to train/eval/test data as well as config files to replicate their results however when I test, I notice that the output from the network has odd float values ie. -7.0234. I assumed the output would be a mask, [0,1] as it is a binary segmentation. Is there an operation I have to perform on the data to get it in correct mask format?<br>
The following is included in the config code so I thought that would handle it and give me proper mask values?</p>
<pre><code class="lang-auto">  final_sigmoid: true
  is_segmentation: true
</code></pre>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a> (Tagging because you have helped so many times when I reply and reading your replies to others haha)</p>
<p>Any help appreciated,</p>
<p>Kyle</p>

ptrblck: <p>Based on <a href="https://github.com/wolny/pytorch-3dunet/blob/62e10674c5cf9f44e252297203213b8c7f23c5f7/pytorch3dunet/unet3d/model.py#L135-L136">this comment</a> from the repository, it seems the final activations are only used during prediction, not training:</p>
<blockquote>
<p>apply final_activation (i.e. Sigmoid or Softmax) only during prediction. During training the network outputs logits and it’s up to the user to normalize it before visualising with tensorboard or computing validation metric</p>
</blockquote>
<p>so the observed values seem reasonable for logits. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

krioux5: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="93089">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>ved values seem reasonable for logits.</p>
</blockquote>
</aside>
<p>To clarify, I meant that when I run an example through prediction, the output is still those float values. ie. I run the command</p>
<pre><code class="lang-auto">predict3dunet --config &lt;CONFIG&gt;
</code></pre>
<p>And the predicted mask contains values such as -7.0412 etc.</p>

ptrblck: <p>In that case either <code>self.testing</code> is still set to <code>False</code> (it seems to be set during initialization and I don’t know, why the internal <code>self.training</code> argument isn’t used) or <code>self.final_activation</code> is <code>None</code>.<br>
Could you check both of these attributes and make sure they are set appropriately?</p>
Google Colab raised an issue Regarding Pytorch

Ganga: <p>Here is my Error:<br>
RuntimeError: isDifferentiableType(variable.scalar_type()) INTERNAL ASSERT FAILED at “/pytorch/torch/csrc/autograd/functions/utils.h”:59, please report a bug to PyTorch.</p>

ptrblck: <p>Could you post a code snippet to reproduce this error or link to your notebook, so that we could have a look?</p>

Ganga: <p>here is my code</p>
<pre><code class="lang-auto">import torch.nn as nn
import csv
import numpy
import numpy as np
class PrintLayer(nn.Module):
    def __init__(self):
        super(PrintLayer, self).__init__()
        m = nn.Conv2d(3, 96, kernel_size=(11,11), stride=(4,4), padding=0, bias = False)
    
    def forward(self, x):
        # Do your print / debug stuff here
        #print("x is",x)
        return x

def init_weights(m):
      if type(m) == nn.Conv2d:
        torch.nn.init.xavier_uniform(m.weight)
        #for weight
        a_file1 = open("1st_layer_8bit_weight.txt","r")
        list_of_lists1 = []
        for line in a_file1:
          stripped_line1 = line.strip()
          line_list1 = stripped_line1.split()
          list_of_lists1.extend(line_list1)
        a_file1.close()

        b = torch.from_numpy(numpy.array(list_of_lists1,dtype= 'int8'))
        w = torch.tensor(b)
        ten = w.reshape(96,3,11,11)
        m.weight.data = ten

        #for bias
        """a_file12 = open("bias.txt","r")
        list_of_lists12 = []
        for line in a_file12:
          stripped_line12 = line.strip()
          line_list12 = stripped_line12.split()
          list_of_lists12.extend(line_list12)
        a_file12.close()

        c = torch.from_numpy(numpy.array(list_of_lists12,dtype= 'float32'))
        d = torch.tensor(c)
        ten1 = d.reshape(256)
        m.bias.data = ten1"""
       
        #print("actual weight is",m.weight)
        #print(m.bias)
      
model = nn.Sequential(
        nn.Conv2d(3, 96, kernel_size=(11,11), stride=(4,4), padding=0, bias = False),
        PrintLayer(),
         # Add Print layer for debug
        )
model.apply(init_weights)

#for input
a_file = open("1st_layer_8bit_input.txt","r")
list_of_lists = []
for line in a_file:
  stripped_line = line.strip()
  line_list = stripped_line.split()
  list_of_lists.extend(line_list)
a_file.close()
b = torch.from_numpy(numpy.array(list_of_lists,dtype= 'int8'))
x = torch.tensor(b)
x = x.reshape(1,3,227,227)
output = model(x)

np.set_printoptions(threshold=50000000,formatter={'float_kind':'{:f}'.format})
output = output.detach().numpy()
file = open("correct_output.txt","w")
file.write(str(output))
file.close()
print(model)
#for weight and bias
print(list(model.named_parameters()))
#print(model.state_dict())```</code></pre>

ptrblck: <p>Thanks for the code. Unfortunately, I cannot reproduce this issue using random input data, since the used file is undefined.<br>
Are you only seeing the error using the weights loaded from the file or could you post an executable code snippet?</p>
Model computation time measurement when using Torchscript in C++

Jungmo_Ahn: <p>Hello</p>
<p>In order to measure the computation time of the deep learning model when using GPU, we have to keep in mind about GPU operations asynchronicity. Thus, we should use torch.cuda.Event or torch.autograd.profiler in Python.</p>
<p>I found torch::autograd::profiler in libtorch but there is no API documentation to know how to use functions in the library.  This looks like C++ version of torch.autograd.profiler which to measure model computation time. However, there are many examples to measure computation time in Python but it is hard to find any example to measure computation time when using GPU in C++.</p>
<p>(This might be because the majority of users except me are familiar to use C++ and C++ library. <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=9" title=":sweat_smile:" class="emoji" alt=":sweat_smile:">)</p>
<p><strong>Could someone provide a simple example to measure computation time of Torchscript model considering both CPU time and GPU time in C++?</strong></p>
<p>FYI, in order to correctly measure computation time in Python3, please refer this link:<br>
<a href="https://discuss.pytorch.org/t/how-to-measure-time-in-pytorch/26964">https://discuss.pytorch.org/t/how-to-measure-time-in-pytorch/26964</a></p>
<p>Many thanks!</p>

glaringlee: <p><a class="mention" href="/u/jungmo_ahn">@Jungmo_Ahn</a><br>
I think you probably should category this as ‘jit’</p>
Passing model, dataset, optimizer etc. into DDP

Rizhiy: <p>Hi, currently I have the following train function:</p>
<pre><code class="lang-python">def train(model, dataset, sampler, criterion, optimizer, scheduler, cfg):
    model = DataParallel(model.cuda())

    loader = DataLoader(dataset, bs=cfg.BS, num_workers=4, sampler=sampler)

    for epoch_idx in range(cfg.EPOCHS):
        for batch, targets in loader:
            preds = model(batch)
            loss = criterion(preds, targets)
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
            scheduler.step()
</code></pre>
<p>How can I convert this code to use <code>DistributeDataParallel</code>?</p>
<p>I looked in the tutorials, but they initialise model, dataset, etc. inside the train function. I can’t do that since I want the signature to remain the same and have the flexibility of defining model, dataset, etc. outside the train function.</p>
<p>Can I just pass all that using arguments in <code>mp.spawn</code>?</p>

mrshenli: <p>Hey <a class="mention" href="/u/rizhiy">@Rizhiy</a></p>
<p>You might not be able to pass the optimizer as that, because every subprocess needs its own dedicated optimizer. And not sure about how dateset/criterion/scheduler in the code behave in multiprocessing use cases. If it is just the model, the following might work.</p>
<pre><code class="lang-python">def train(rank, model):
    model = DistributedDataParallel(model.to(rank), device_ids=[rank], output_device=rank)
    ...

def main():
    model.share_memory() 
    mp.spawn(
        train, 
        args=(model, dataset, ...),
        nprocs=world_size,
        join=True)

if __name__=="__main__":
    main()
</code></pre>
<p>Rank is the subprocess id, which is provided my <code>mp.spawn</code> as the first argument to the target function.</p>
<p>If the reason for this is to keep the <code>train()</code> signature intact, is it possible to create another <code>wrapper</code> function to wrap <code>train()</code>, configure everything in <code>wrapper</code>, call <code>train</code> in <code>wrapper</code>, and use <code>wrapper</code> as the spawn target?</p>

Rizhiy: <p>Hi <a class="mention" href="/u/mrshenli">@mrshenli</a>, the reason is that this function is part of the internal framework and other programmers should be able to setup the arguments how they require and <code>train()</code> is only responsible for the loop.</p>

Rizhiy: <p>I guess my main question is: what can be passed as arguments in <code>mp.spawn()</code>?</p>

mrshenli: <p>IIUC, it can accept shared memory tensor and anything that’s pickable with Python multiprocessing</p>
How to use the system nccl library instead of the internal one of pytorch when using ddp?

ayl: <p>I have installed pytorch by using conda and I can directly use nccl backend tro do distributed training.  However, the internal nccl library of pytorch is 2.4.8. If I want use another manually installed nccl library such as 2.7.8 version, how can I do it? Is there any way without compiling pytroch from souce?</p>

mrshenli: <p>Hey <a class="mention" href="/u/ayl">@ayl</a></p>
<p>You can <code>export USE_SYSTEM_NCCL=1</code>, and then compile PyTorch from source.</p>
<p>See this discussion <a href="https://discuss.pytorch.org/t/torch-distributed-not-working-on-two-machines-nccl-backend/87659" class="inline-onebox">Torch distributed not working on two machines [nccl backend]</a></p>

ayl: <aside class="quote no-group" data-username="ayl" data-post="1" data-topic="93198">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/b3f665/48.png" class="avatar"> ayl:</div>
<blockquote>
<p>Is there any way without compiling pytroch from souce?</p>
</blockquote>
</aside>
<p>Thank you. Is there any way without compiling pytorch from source?</p>

mrshenli: <p>I don’t think there is an easy/safe way to do so, as the NCCL API also changes from release to release. Even if you can dynamically link libnccl, it might not be compatible with the built libtorch.</p>
How to freeze custom learnable weight?

wonchulSon: <p>I add <code>my_weight </code> to optimizer like below.</p>
<pre><code class="lang-auto">my_weight = nn.Parameter(torch.tensor(args.weight).cuda(), requires_grad=True)
self.optimizer.add_param_group({"params": [my_weight]})
</code></pre>
<p>And I want freeze when the <code>iteration</code> is 3 times.</p>
<pre><code class="lang-auto">for batch_idx, (data, target) in enumerate(train_loader):
    iteration += 1
    optimizer.zero_grad()
    output = model(data)
    loss = criterion(output, target)

    if iteration % 3 == 0:
        !!!!! I want freeze my_weight here !!!!!
    else:
        !!!!! I want unfreeze my_weight here !!!!!

    loss.backward()
    optimizer.step()
</code></pre>
<p>How can I do?</p>
<p>Thank you.</p>

user_123454321: <p>I did something like this, but definitely not very elegant…</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; import torch.nn as nn
&gt;&gt;&gt; 
&gt;&gt;&gt; device = 'cuda:0'
&gt;&gt;&gt; 
&gt;&gt;&gt; w = nn.Parameter(torch.randn(1).to(device), requires_grad=True)
&gt;&gt;&gt; b = nn.Parameter(torch.randn(1).to(device), requires_grad=True)
&gt;&gt;&gt; 
&gt;&gt;&gt; opt = torch.optim.Adam([w], 0.1)
&gt;&gt;&gt; opt.add_param_group({'params': [b]})
&gt;&gt;&gt; 
&gt;&gt;&gt; for i in range(9):
...     opt.zero_grad()
...     x = torch.randn(10, 1).to(device)
...     y = 3.14 * x + 1.85
...     y_pred = x @ w + b
...     loss = ((y_pred - y) ** 2).mean()
...     loss.backward()
...     opt.step()
...     if i % 3 == 0:
...             b.requires_grad = False
...             del opt.param_groups[1] # second param group as added it later, I know pretty inelegant
...             del opt.state[b]
...     elif not b.requires_grad:
...             b.requires_grad = True
...             opt.add_param_group({'params': [b]})
...     print('Iteration: {}, w: {}, b: {}'.format(i, w.item(), b.item()))
... 
Iteration: 0, w: 0.812088131904602, b: 1.4590768814086914
Iteration: 1, w: 0.7677664160728455, b: 1.4590768814086914
Iteration: 2, w: 0.7146307229995728, b: 1.5590769052505493
Iteration: 3, w: 0.6447014808654785, b: 1.611872673034668
Iteration: 4, w: 0.5945112705230713, b: 1.611872673034668
Iteration: 5, w: 0.5443614721298218, b: 1.51187264919281
Iteration: 6, w: 0.48854923248291016, b: 1.5435525178909302
Iteration: 7, w: 0.42736998200416565, b: 1.5435525178909302
Iteration: 8, w: 0.3671201169490814, b: 1.643552541732788

</code></pre>
Monte Carlo Sampling: CUDA Out of memory

mflova: <p>Hi!<br>
I am trying to perform Monte Carlo sampling to estimate the uncertainty of my network. To do so, I have dropout enabled and I am executing following loop:</p>
<pre><code class="lang-auto">outputs = [self._model(inputs) for i in range(X)]
</code></pre>
<p>Where “inputs” is just a single frame.<br>
First I tried this code in Google Colab, and everything was working fine, even with X = 40 and batch_size = 100 (frames) at once.<br>
When I switch back to a personal computer with a 6GB dedicated memory GPU, if X is greater than 5 (with one single frame), I get the famous error “CUDA Out of memory”. Even if I split the code such that:</p>
<pre><code class="lang-auto">outputs = [self._model(inputs) for i in range(X/2)]
outputs = [self._model(inputs) for i in range(X/2)]
</code></pre>
<p>The code gives me same results.<br>
Why is this happening? Is the for loop being parallelized by CUDA API and thus collapsing the memory? How can I solve this?</p>
<p>Thank you in advance</p>

mapostig: <p>Hello,</p>
<p>I don’t know if this can solve your issue. But I was having a similar one (i am looping a model over different folds) and at the second loop I obtained this famous memory error.<br>
At the end of the loop I free the memory in the following way:</p>
<blockquote>
<p>for element in dir():</p>
<pre><code>    if element[0:2] != "__":

        del globals()[element]

import torch

torch.cuda.empty_cache()
</code></pre>
</blockquote>
<p>I found this solution on <a href="https://www.kite.com/python/answers/how-to-delete-variables-and-functions-from-memory-in-python" class="inline-onebox" rel="noopener nofollow ugc">Kite</a></p>
<p>I hope it solves your problem</p>

mflova: <p>First of all, thank you for your response.<br>
I have tried your approach. However, I still need so many variables in my script, so I cannot do that, as the programm wouldn’t work as a consequence. If I only use “torch.cuda.empty_cache()”, I still get “CUDA out of memory”, so aparently I cannot solve it with that.</p>
<p>EDIT:<br>
This crashes:</p>
<pre><code class="lang-auto">        outputs = self._model(inputs)
        outputs2 = self._model(inputs)
        outputs3 = self._model(inputs)
        outputs4 = self._model(inputs)
        outputs5 = self._model(inputs)
        outputs6 = self._model(inputs)
        outputs7 = self._model(inputs)
        outputs8 = self._model(inputs)
        outputs9 = self._model(inputs)
</code></pre>
<p>While this not:</p>
<pre><code class="lang-auto">        outputs = self._model(inputs)
        outputs = self._model(inputs)
        outputs = self._model(inputs)
        outputs = self._model(inputs)
        outputs = self._model(inputs)
        outputs = self._model(inputs)
        outputs = self._model(inputs)
        outputs = self._model(inputs)
        outputs = self._model(inputs)
</code></pre>
<p>So it is clearly about the memory used to allocate the output tensors in the GPU. I have tried to first replicate the outputs in a CPU tensor before computing more samples, but it is still not working.</p>
How to use 2gpus to train in single machine (node)?

Hao_Meng: <p>Hi, everyone!<br>
Recently I want to train a network using 2 gpus on 1 machine(node).But I really get confused cause I can not find one which descrip train ,validiation ,saving checkpoints and load checkpoints concretely?<br>
So is there any great example could help?</p>

mrshenli: <p>Here is the overview for the distributed training tools offered by PyTorch: <a href="https://pytorch.org/tutorials/beginner/dist_overview.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/dist_overview.html</a></p>
<p>If you are looking for data parallel training, you might want to start from <code>DataParallel</code>?</p>

Hao_Meng: <p>Thanks， I found tutorial w.r.t. DataParallel is much easier to understand and implement.<br>
However, using <code>torch.nn.parallel.DistributedDataParallel</code> is much more difficult.</p>

mrshenli: <p>Yep, <code>DataParallel</code> indeed is an easier entry point, but is not the most efficient solution. If you are looking for faster training speed or scaling to more machines later, DDP would still be the way to go.</p>

Hao_Meng: <p>Tanks, I find the example code of DDP on Imagenet is not easy to imitate to fit my code. Is there any more detailed  example?</p>

osalpekar: <p>Here are some more general DDP examples/tutorials:</p>
<aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="16" height="16">
      <a href="https://github.com/pytorch/examples/tree/master/distributed/ddp" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars2.githubusercontent.com/u/21003710?s=400&amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/pytorch/examples/tree/master/distributed/ddp" target="_blank" rel="nofollow noopener">pytorch/examples</a></h3>

<p>A set of examples around pytorch in Vision, Text, Reinforcement Learning, etc. - pytorch/examples</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p><a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/intermediate/ddp_tutorial.html</a></p>
<p>There are also several example projects of varying complexity on GitHub that use DistributedDataParallel. They would be a great reference for pytorch code across a variety of domains.</p>

mrshenli: <p>Besides the link <a class="mention" href="/u/osalpekar">@osalpekar</a> posted above, here is a summary of all DDP docs we have currently: <a href="https://pytorch.org/tutorials/beginner/dist_overview.html#torch-nn-parallel-distributeddataparallel" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/dist_overview.html#torch-nn-parallel-distributeddataparallel</a></p>

Hao_Meng: <p>Thanks a lot. I will have a try.</p>

Hao_Meng: <p>Thanks. I will have a try.</p>
Finding the nearest matching descriptor

Mr_Randy_Tom: <p>I’m using the following code to retrieve the nearest matching descriptors index and value.</p>
<blockquote>
<p>def <strong>approx</strong>(self, db_desc, query_desc, n_neighbors):</p>
<pre><code>    db_features = __to_unit_torch__(db_desc, cuda="cuda")
    query__desc = __to_unit_torch__(query_desc, cuda="cuda")
    with torch.no_grad():
        d = 1. - torch.matmul(db_features, query__desc.transpose(0,1))
        values, indices = torch.topk(d, n_neighbors, dim=1, largest=False, sorted=True)
        return indices.cpu().numpy()
</code></pre>
</blockquote>
<p>why is that it always returns 0?</p>
Freeze_module pass for 1.4.x version of PyTorch

vdantu: <p>I see that there is a freeze_module pass in 1.5.x and later in PyTorch.</p>
<p>Is there a similar pass in 1.4.x version of PyTorch which inlines the graph and resolves the weights and biases in the PyTorch graph into constants?</p>
<p>Does running <code>_jit_pass_lower_graph()</code>, and converting all the tensor parameters returned by this pass into constants in the graph have the same effect?</p>

zdevito: <p>The <code>freeze_module</code> pass was added to 1.5, so there isn’t anything 100% equivalent in 1.4 The <code>_jit_pass_lower_graph</code> does something similar but freezing goes beyond it to try to simplify the graph based on the changes, and to verify that the conversion to constants was a correct transform.</p>

vdantu: <p><a class="mention" href="/u/zdevito">@zdevito</a> :  Thanks for the reply. If I had to achieve something similar to <code>freeze_module</code> in PyTorch 1.4.x, is running <code>_jit_pass_lower_graph()</code> and converting the returned parameters into graph constants a good approach? Or is there a better mechanism to achieve this? Porting <code>freeze_module</code>  to 1.4.x seems cumbersome as it relies on AliasDB APIs which don’t exist in 1.4.x.</p>
How to feed two dimension tensor to Multi-head attention?

Thomas_34: <p>I am working with two dimension input</p>
<p>Batch_size = 23<br>
features      = 34</p>
<p><code>Ex :  data = np.random.uniform(-1,1,[batch_size, features ])</code></p>
<p>I want to apply Multi-head attention on that but Multi-head attention accepts input <code>[  batch_size. x sequence_length x embedding_dim ]</code></p>
<p>I don’t want to use embeddings before attention to make <code>two dim =&gt;Embedding =&gt; three dim</code> as we do in LSTM.</p>
<p>What are the ways we can feed this input to Multihead attention?<br>
Expanding a new dim in existing data is one of the solutions, should I expand dim at  1st axis (<code>np.expand_dims(data, axis=1)</code>) or 2nd axis (<code>np.expand_dims(data, axis=2)</code>)?</p>
<p>Thank you!</p>
How to balance GPU memories in DDP?

jaehyung.ca: <p>When training a model with DDP, GPU for rank 0 consumes much higher memory than others.<br>
Because of that GPU, I cannot increase batch-size for training.<br>
Is there a good way to deal it with?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/7/87e30529ae70d263e4680eac8b99629a49d912cb.png" data-download-href="https://discuss.pytorch.org/uploads/default/87e30529ae70d263e4680eac8b99629a49d912cb" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/7/87e30529ae70d263e4680eac8b99629a49d912cb_2_575x500.png" alt="image" data-base62-sha1="jo6QvM2B6em3uaenyZyfEAWIJbJ" width="575" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/7/87e30529ae70d263e4680eac8b99629a49d912cb_2_575x500.png, https://discuss.pytorch.org/uploads/default/original/3X/8/7/87e30529ae70d263e4680eac8b99629a49d912cb.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/7/87e30529ae70d263e4680eac8b99629a49d912cb.png 2x" data-dominant-color="3A1830"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">794×690 88.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

mrshenli: <p>Hey <a class="mention" href="/u/jaehyung.ca">@jaehyung.ca</a> did you intentionally create any tensor on <code>cuda:0</code> from every process? If not, it might be some lib/code accidentally create states on <code>cuda:0</code>. To avoid this, you can set <code>CUDA_VISIBLE_DEVICES</code> env var to make sure that all processes only sees one GPU.</p>

jaehyung.ca: <p>Thank for the reply <a class="mention" href="/u/mrshenli">@mrshenli</a></p>
<p>I haven’t set explicitly device cuda:0 at any point.<br>
And even in the <a href="https://github.com/pytorch/examples/blob/master/distributed/ddp/main.py" rel="nofollow noopener">official DDP example code</a> shows the same unbalanced GPU memory consumption.</p>
<p>I solved the issue by setting torch.cuda.set_device(args.local_rank) which works the same as setting CUDA_VISIBLE_DEVICES.</p>
Munmap_chunk(): invalid pointer

peijason: <p>I can successfully run  <strong>yolov5</strong>   <code>python detect.py</code>  to carry out the test.</p>
<p>However, whenever I created another file, say,  <code>myfile.py</code> , and trying to packed it into a  <strong>class</strong> ,  <code>device = select_device(opt.device)</code>  will fail with this  <strong>ERROR</strong>  message:  <code>munmap_chunk(): invalid pointer</code> .</p>
<p>Please help…</p>

peijason: <p>Problem found…<br>
It looks my <strong>PyTorch</strong> is <strong>NOT</strong> successfully built against <strong>CUDA</strong> ? hmmmmm…</p>
All reducing tensors

vineeths: <p>Suppose I have a vector of type <code>torch.int32</code>. During all reduce operation, do all 32bits for each coordinate gets transmitted, irrespective of the value (at the coordinate)?</p>
<p>More specifically, I am interested in how we achieve higher speeds in reducing sparse tensors. (By sparse tensors I mean tensors wil large number of zeroes).</p>

osalpekar: <p>We currently support <code>all_reduce</code> on sparse tensors with the Gloo backend (for both CPU and CUDA tensors), but this is not yet supported with the NCCL backend.</p>

vineeths: <p>Tensorflow <code>all_reduce</code>s sparse tensors (tf.IndexedSlices), by <code>all_gather</code>ing followed by tensor reduction.</p>
<p>Does PyTorch do the same (with Gloo backend) or it does something different under the hood?</p>

osalpekar: <p>It’s pretty similar - we <code>all_gather</code> the metadata, indices, and values, and then each node does a local sum of the sparse tensors. Here’s the implementation:  <a href="https://github.com/pytorch/pytorch/blob/65bd38127a34d428915c88507878b4735edf005f/torch/lib/c10d/ProcessGroupGloo.cpp#L939" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/65bd38127a34d428915c88507878b4735edf005f/torch/lib/c10d/ProcessGroupGloo.cpp#L939</a></p>