akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to perform inference with nn.Transformer

username: steve365
cooked: <p>I trained a Transformer model using the default <a href="https://pytorch.org/docs/stable/generated/torch.nn.Transformer.html" rel="noopener nofollow ugc">nn.Transformer class</a> to perform machine translation.</p>
<p>During training, we pass both the inputs into the encoder and the targets into the decoder. In pseudocode, a forward pass looks like:</p>
<pre><code class="lang-auto">for (inputs, targets) in train_loader:
    preds = transformer(src=inputs, tgt=targets)
</code></pre>
<p>My question is: what do we do with the <code>tgt</code> argument at test time? I have tried:</p>
<ul>
<li>Feed in a boolean <code>src_mask</code> with <code>True</code> at each position</li>
<li>Feed in a <code>tgt = torch.zeros_like(targets)</code>
</li>
</ul>
<p>But these decrease the model performance.</p>
<p>Thank you for your help (:</p>

username: rravu3
cooked: <p>At inference time, you are essentially generating the text. So, you will have to provide the currently generated text (beginning with start token) and then either greedily generate the next token or maintain a list of hypotheses.</p>
<p>Also, I am guessing you are providing more than just the input and target in your full code to account for masking (of keys associated with padding, looking at currently decoded positions, etc).</p>
Error related to the kernel size and padding for a conv2D

username: Tm4
cooked: <p>I have a tensor with the size of <code>(batch_size= 16,channels= 192,H=7,W=7)</code>. I want to decrease the number of channels to 160 channels but having the same height and width. The <code>my</code> class is the main class that I use for running the experiment and the <code>x</code> (the mentioned tensor), is the input to the <code>my</code> class:</p>
<pre><code>class MyConv2d(nn.Module):
def __init__(self, in_planes, out_planes, kernel_size, stride, padding=0):
    super(MyConv2d, self).__init__()
    self.conv = nn.Conv2d(in_planes, out_planes, kernel_size=kernel_size, stride=stride, padding=padding, bias=False)
    self.bn = nn.BatchNorm2d(out_planes, eps=1e-3, momentum=0.001, affine=True)
    self.relu = nn.ReLU()

def forward(self, x):
    x = self.conv(x)    # The error happens here
    x = self.bn(x)
    x = self.relu(x)
    return x

class my(nn.Module):
    def __init__(self, in_channel=192, out_channel=160, out_sigmoid=False):
        super(my, self).__init__()
        
        self.out_sigmoid=out_sigmoid
        
        self.deconv = self._make_deconv(in_channel, out_channel)
     
    def _make_deconv(self, in_channel, out_channel, kernel_size=3, stride=1, padding=1):
        layers=[]
        layers.append(MyConv2d(in_channel, out_channel ,kernel_size=kernel_size, stride=stride, padding=padding))
        
        return nn.Sequential(*layers)
    
    def forward(self, x):
        x=self.deconv(x)
        return x
</code></pre>
<p>Unfortunately I face with the following error:<br>
<a href="https://i.stack.imgur.com/zMQWP.jpg" rel="noopener nofollow ugc"><img src="https://i.stack.imgur.com/zMQWP.jpg" alt="" width="690" height="63"></a></p>
<p>How can I fix the problem?</p>

username: ptrblck
cooked: <p>The error message doesn’t point to the kernel size and padding, but to a mismatch in the number of input channels. Your current input seems to have <code>192</code> channels while <code>768</code> are expected. Use <code>in_planes=192</code> and it should work.</p>
Triplet loss for semantic segmentation

username: avatar
cooked: <p>Hi, I was trying to use the triplet loss function in order to train a semantic segmentation model.</p>
<pre><code>class TripletLoss(nn.Module):
   def __init__(self, margin=None):
     super(TripletLoss, self).__init__()
     self.margin = margin
     if self.margin is None:  
        self.Loss = nn.SoftMarginLoss()
     else:
        self.Loss = nn.TripletMarginLoss(margin=margin, p=2)

   def forward(self, anchor, pos, neg):
     if self.margin is None:
        num_samples = anchor.shape[0]
        y = torch.ones((num_samples, 1)).view(-1)
        if anchor.is_cuda: y = y.cuda()
        ap_dist = torch.norm(anchor-pos, 2, dim=1).view(-1)
        an_dist = torch.norm(anchor-neg, 2, dim=1).view(-1)
        loss = self.Loss(an_dist - ap_dist, y)
     else:
        loss = self.Loss(anchor, pos, neg)

     return loss
</code></pre>
<p>My question is how I can sample triplets of pixel embeddings for the calculation of this loss. Does anyone know if there is an implementation of a pixel-wise triplet loss for dense tasks such image segmentation?</p>
When a model is imported (via torch.load) into a training file it does not learn, but when initialized in same file does

username: MicBlizzard
cooked: <p>Hopefully the title makes sense (I am relatively new to python and pytorch).</p>
<p>Basically, I have two python files that I use to create a model (a nn.Module sub-class), which I then fully save (not the dictionary of parameters, but the entire model) using the torch.save(model,PATH_OF_PTH) method in a pth file. I then load the model using torch.load(PATH_OF_PTH) in my train.py file and do the usual training (using nn.CrossEntropyLoss() as my loss and optim.SGD(model.parameters(), lr=0.001, momentum=0.9) as my optimizer). With this set-up my loss decreases very slowly and the model barely learns.</p>
<p>However, if I copy paste my model generating code into my training file and do not save and load the model, but directly train it - the training is a lot more efficient and the loss decreases a lot faster.</p>
<p>Does anyone know what could be the cause of this?</p>
<p>Thank you in advance!</p>

username: ptrblck
cooked: <p>Using <code>torch.save</code> and <code>torch.load</code> would load the entire model and all its dependencies. I don’t know which methods are used but are some maybe overridden? I would probably try to compare the outputs of both approaches using static inputs (e.g. <code>torch.ones</code>) and in <code>model.eval()</code> mode to check if the model itself is returning different values (maybe also different parameter sets are used).</p>
Is embedding layer different from linear layer

username: Rexedoziem
cooked: <p>What is the difference between embedding layers and linear layers, and how it works in nlp?</p>

username: ptrblck
cooked: <p>Embedding layers are used as trainable “lookup” tables using the input as indices, while linear layers apply a matrix multiplication using the input and the internal trainable <code>weight</code> parameter and add the <code>bias</code> to the result afterwards.</p>

username: Rexedoziem
cooked: <p>That means I can place the embedded layers in the model and output it through the linear layer?</p>

username: ptrblck
cooked: <p>Yes, you can use the output of embedding layers in linear layers as seen here:</p>
<pre><code class="lang-python">num_embeddings = 10
embedding_dim= 100
emb = nn.Embedding(num_embeddings, embedding_dim)

output_dim = 5
lin = nn.Linear(embedding_dim, output_dim)

batch_size = 2
x = torch.randint(0, num_embeddings, (batch_size,))

out = emb(x)
print(out.shape)
# torch.Size([2, 100])

out = lin(out)
print(out.shape)
# torch.Size([2, 5])
</code></pre>

username: Rexedoziem
cooked: <p>Thanks, but after placing it in the model, work with multioutput classification, I needed to output 6 target variables using nn.sequential in Bert model, but received an error saying nn.sequential doesn’t work with multiple inputs?</p>

username: ptrblck
cooked: <p>That’s correct. <code>nn.Sequential</code> expects one input and will yield one output.<br>
You can check other topics discussing this feature (e.g. <a href="https://discuss.pytorch.org/t/how-can-i-pass-multiple-inputs-to-nn-sequential-layers/50868">here</a>) which describe some workarounds.</p>

username: Rexedoziem
cooked: <p>Then I can just do it without nn.sequential</p>

username: Rexedoziem
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/c/5c0f02a4ed48d1b997c6e1f9d56c409a3ebde805.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/5c0f02a4ed48d1b997c6e1f9d56c409a3ebde805" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/c/5c0f02a4ed48d1b997c6e1f9d56c409a3ebde805_2_681x500.jpeg" alt="image" data-base62-sha1="d8o5nw46MxuQY1kj9lr0OS2VOlf" width="681" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/c/5c0f02a4ed48d1b997c6e1f9d56c409a3ebde805_2_681x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/5/c/5c0f02a4ed48d1b997c6e1f9d56c409a3ebde805_2_1021x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/c/5c0f02a4ed48d1b997c6e1f9d56c409a3ebde805_2_1362x1000.jpeg 2x" data-dominant-color="4C4E51"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1920×1409 144 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Just in this form right?</p>
Unable to select GPU

username: robotechnics
cooked: <p>Hi,<br>
I have 10 GPUs available and 1 GPU is in used by another torch process. I would like to run another process on any of the remaining GPUs but I always get the error message:<br>
RuntimeError: CUDA error: all CUDA-capable devices are busy or unavailable</p>
<p>I tried several options and none of them worked:</p>
<ol>
<li>Selecting GPUs on python script<br>
import os<br>
os.environ[“CUDA_DEVICE_ORDER”]    = ‘PCI_BUS_ID’<br>
os.environ[“CUDA_VISIBLE_DEVICES”] = ‘0,1,3,4’<br>
print(f’[INFO] Using GPU: {torch.cuda.current_device()}‘)<br>
print(f’[INFO] Available GPUs: {torch.cuda.device_count()}')</li>
</ol>
<h1><a name="h-1" class="anchor" href="#h-1"></a></h1>
<p>for d in range(torch.cuda.device_count()):<br>
print(torch.cuda.get_device_name(d))</p>
<p>But It only recognise 1 GPU:<br>
[INFO] Using GPU: 0<br>
[INFO] Available GPUs: 1<br>
GeForce GTX 1080 Ti</p>
<ol start="2">
<li>Selecting GPU on command line:<br>
CUDA_VISIBLE_DEVICES=0,1,3,4 python gdxray_cganTrainer.py</li>
</ol>
<p>and I check on linux using<br>
env | grep CUDA_VISIBLE_DEVICES<br>
 →  CUDA_VISIBLE_DEVICES=1,3,4</p>
<p>None of the above 2 options work</p>
<p>I am using torch 1.8.0 and python 3.8.8</p>
<p>Any help?</p>
References/classification/train.py not learning (version v0.12.0)

username: jpcbertoldo
cooked: <p>I am trying to train a <code>resnet18</code> on imagenet using the recipe from torch/vision but it is not learning.</p>
<p>I am launching the script with <code>torchrun references/classification/train.py</code> (version v0.12.0) with the recommended hyperparameters from the <a href="https://github.com/pytorch/vision/blob/9b5a3fecc72434dbd65148723efe54b28b9728c9/references/classification/README.md#resnet" rel="noopener nofollow ugc">readme</a>, except for:</p>
<ul>
<li>the number of GPUs (they recommend 8 but I only have 4)</li>
<li>batch size (which I increased from 32 to 64 to compensate the number of GPUs);</li>
<li>i added <code>--use-deterministic-algorithms</code></li>
</ul>
<p>Result:</p>
<pre><code class="lang-auto">torchrun \
    --nproc_per_node=4 \
    train.py \ 
    --workers 4 \ 
    --batch-size 64 \ 
    --lr 0.1 \ 
    --momentum 0.9 \ 
    --weight-decay 1e-4 \ 
    --lr-step-size 30 \ 
    --lr-gamma 0.1 \ 
    --epochs 90 \ 
    --model resnet18 \ 
    --cache-dataset \ 
    --use-deterministic-algorithms \ 
    --data-path /.../imagenet 
</code></pre>
<p>In fact, I made minor modifications in the script to fix a manual seed and send logs to my wandb, but I belive that shouldn’t have such impact. Here is the <a href="https://github.com/jpcbertoldo/torchvision/pull/4/files" rel="noopener nofollow ugc">diff</a>.</p>
<hr>
<p>I believe these changes shouldn’t have impacted so much, right?</p>
<p>But the model doesn’t seem to be learning but just overfitting. Here are the training curves for the accu1, and accu5, on the train and val sets.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/d/2da776b1aee748a14709c1bbf2786eb401ac409f.png" data-download-href="https://discuss.pytorch.org/uploads/default/2da776b1aee748a14709c1bbf2786eb401ac409f" title="W&amp;B Chart 9_26_2022, 11_20_43 AM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/d/2da776b1aee748a14709c1bbf2786eb401ac409f_2_690x362.png" alt="W&amp;B Chart 9_26_2022, 11_20_43 AM" data-base62-sha1="6vSh4zsblRjdBGSAMtcUM2PLCmX" width="690" height="362" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/d/2da776b1aee748a14709c1bbf2786eb401ac409f_2_690x362.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/d/2da776b1aee748a14709c1bbf2786eb401ac409f_2_1035x543.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/d/2da776b1aee748a14709c1bbf2786eb401ac409f_2_1380x724.png 2x" data-dominant-color="FCFCFC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">W&amp;B Chart 9_26_2022, 11_20_43 AM</span><span class="informations">2528×1328 213 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
How can I get save_mean and save_var from cudnn_batch_norm?

username: Evan_S
cooked: <p>I need mean and variance values to do some operation. So I would like to get the mean and the variance values computed by BatchNorm2d on forward pass, as computing it twice is not ideal for me as evaluation time is extremely crucial in my usage.<br>
Looking into the source code I noticed that cudnnBatchNormalizationForwardTraining function outputs the mean and the variance it computed. The question is, where are these values stored, and how can I access these values?</p>
How to select topk element from similarity matrix

username: Corn_Chang
cooked: <p>I am currently using top-k to select elements from the similarity matrix<br>
So for each x, we can find k elements from y.<br>
How could I use the indices to get something like the below without loops.</p>
<pre><code class="lang-auto">import torch
import torch.nn.functional as F
x = torch.randn((10,64,512)) # (b, x_c, d)
y = torch.randn((10,128,512)) # (b, y_c, d)
sim_matrix = torch.matmul(F.normalize(x, p=2, dim=-1), F.normalize(y, p=2, dim=-1).permute(0, 2, 1)) #  (b, x_c, y_c)
_, sim_matrix_ind = sim_matrix.topk(5, dim=2) # (b, x_c, 5)
select_y = torch.gather(y, dim=1,)  # Maybe (b , x_c, 5, d)
</code></pre>
Backdrop through eigen-decomposition

username: Jack_Shi
cooked: <p>Part of my model involves an eigen-decomposition, i.e. using <code>torch.linalg.eigh</code>, and I get the following error that I do not understand. Anyone has any ideas on what the error implies?</p>
<p><code>RuntimeError: cusolver error: CUSOLVER_STATUS_INTERNAL_ERROR, when calling cusolverDnXsyevd( handle, params, jobz, uplo, n, CUDA_R_32F, reinterpret_cast&lt;void*&gt;(A), lda, CUDA_R_32F, reinterpret_cast&lt;void*&gt;(W), CUDA_R_32F, reinterpret_cast&lt;void*&gt;(bufferOnDevice), workspaceInBytesOnDevice, reinterpret_cast&lt;void*&gt;(bufferOnHost), workspaceInBytesOnHost, info)</code></p>

username: ptrblck
cooked: <p>Could you post a minimal, executable code snippet which would reproduce the issue in the latest stable/nightly release, please?</p>
Performance gap between `batch_size==32` and `batch_size==8, gradient_accumulation==4`

username: pt_z
cooked: <p>I tried to use gradient accumulation in my project. To my understanding, the gradient accumulation is the same as increasing the batch size by x times. I tried <code>batch_size==32</code> and <code>batch_size==8, gradient_accumulation==4</code> in my project, however the result varies even when I disabled <code>shuffle</code> in dataloader. The <code>batch_size==8, accumulation==4</code> variant’s result is significantly poorer.</p>
<p>I wonder why?</p>
<p>Here is my snippet:</p>
<pre><code class="lang-python">loss = model(x)
epoch_loss += float(loss)

loss.backward()

# step starts from 1
if (step % accumulate_step == 0) or (step == len(dataloader)):

    if clip_grad_norm &gt; 0:
        nn.utils.clip_grad_norm_(model.parameters(), max_norm=clip_grad_norm)

    optimizer.step()
    if scheduler:
        scheduler.step()

    optimizer.zero_grad()
</code></pre>

username: ptrblck
cooked: <aside class="quote no-group" data-username="pt_z" data-post="1" data-topic="162135">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pt_z/48/53489_2.png" class="avatar"> pt_z:</div>
<blockquote>
<p>To my understanding, the gradient accumulation is the same as increasing the batch size by x times.</p>
</blockquote>
</aside>
<p>That should be the case, if you are not using any layers which have a dependency on the batch size, such as batchnorm layers. Are you using these layers, which would explain the different outputs?</p>

username: pt_z
cooked: <p>Hi, I figured it out. See this question: <a href="https://stackoverflow.com/questions/73844065/performance-gap-between-batch-size-32-and-batch-size-8-gradient-accumulati/73845149?noredirect=1#comment130393940_73845149" class="inline-onebox" rel="noopener nofollow ugc">python - Performance gap between `batch_size==32` and `batch_size==8, gradient_accumulation==4` - Stack Overflow</a></p>
<p>Everything is due to the dropout.</p>
How backward works for two dependent forward pass

username: J_B_28
cooked: <p>I have to implement two dependent forward pass, I am wondering how the backward works for this case</p>
<pre><code>    x = F.relu(self.lin1(x))
    x = F.relu(self.lin2(x))
    out1 = self.prop1(x)
    out2 = self.prop2(x)
    out1 = F.log_softmax(out1, dim=1)
    out2 = F.log_softmax(out2, dim=1)
</code></pre>
<p>return out1, out2</p>
<p>out1 and out2 share the same input x, will loss.backward() work for this case? Thanks.</p>

username: ptrblck
cooked: <p><code>loss</code> is undefined in your case, but e.g. if you are summing <code>outX</code> to <code>loss</code>, <code>loss.backward()</code> will work and will use both <code>outX</code> tensors during the backpropagation.</p>

username: J_B_28
cooked: <p>Thanks so much for your reply. I think I can define:<br>
loss1 = F.cross_entropy(out1, label)<br>
loss2 = F.cross_entropy(out2, label)<br>
(loss1+loss2).backward()</p>
<p>I still have a question about back-propagation, if I want to get the gradient of x, I defined a backward hook:<br>
x = F.relu(self.lin1(x))<br>
x = F.relu(self.lin2(x))<br>
def backward_hook(grad):<br>
<span class="hashtag">#some</span> operations<br>
return grad<br>
x.register_hook(backward_hook)<br>
out1 = self.prop1(x)<br>
out2 = self.prop2(x)<br>
out1 = F.log_softmax(out1, dim=1)<br>
out2 = F.log_softmax(out2, dim=1)<br>
return out1, out2</p>
<p>for loss:<br>
loss1 = F.cross_entropy(out1, label)<br>
loss2 = F.cross_entropy(out2, label)<br>
(loss1+loss2).backward()</p>
<p>Will backward prop follow the different forward path to compute the two separate gradient? If so, the grad in the backward hook in the above code is for out1 or out2? Thanks for your help.</p>

username: ptrblck
cooked: <p>I think the gradient hook will be called after the gradient accumulation is done, so if <code>x</code> is used in both outputs, its <code>grad</code> will be calculated from both losses as well.</p>

username: J_B_28
cooked: <p>Thanks for your help. I am wondering if I can get the two gradients of x for both two forward pass separately in that backward hook. In other word, if there is just one forward pass, the grad of x is just for that forward pass, but if I use two forward pass like above, how can I separate and get gradients in the backward hook? Thanks so much.</p>
Do we need to include batchnorm and activation after conv2d transpose?

username: Coca_Cola12
cooked: <p>Hello, just a conceptual question. I see that when we use conv2d it’s usually followed by batchnorm and activation. Is this also true for conv2d transpose? If yes, is it because of the same logic as conv2d transpose?</p>
<p>Goal: I’m trying to stack</p>
<blockquote>
<p>self.up1 = nn.ConvTranspose2d(in_channels=out3, out_channels=out2, kernel_size=3, padding=1,<br>
stride=2, output_padding=1)<br>
self.up2 = nn.ConvTranspose2d(in_channels=out2, out_channels=out1, kernel_size=3, padding=1,<br>
stride=2, output_padding=1)<br>
self.up3 = nn.ConvTranspose2d(in_channels=out1, out_channels=n_out_channels, kernel_size=3, padding=1,stride=2, output_padding=1)</p>
</blockquote>
<p>So I’m just wondering if I need to use both batchnorm and ReLU between those layers.</p>
Is there any source of libTorch without NNPACK?

username: Zhicheng_Wang
cooked: <p>Hi,</p>
<p>I wish to run libtorch for Conv1D layers on a robot. I used the CXX_ABI-12.1-cpu version.  But it got stuck and produced the error of <code>Could not initialize NNPACK! Reason: Unsupported hardware</code>.</p>
<p>I searched for the issue and found that NNPACK requires avx2 instruction set, which might not be supported on my embedding system oriented CPU.</p>
<p>So I am searching for a way to re-compile libTorch without NNPACK. Is that possible? Do I need to replace the precompiled code source or just add some argument like USE_NNPACK=False to CMake? How will the running speed be influenced if I got rid of NNPACK?</p>
<p>Suggestions will be appreciated.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Zhicheng_Wang" data-post="1" data-topic="162146">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zhicheng_wang/48/53493_2.png" class="avatar"> Zhicheng_Wang:</div>
<blockquote>
<p>Do I need to replace the precompiled code source or just add some argument like USE_NNPACK=False to CMake?</p>
</blockquote>
</aside>
<p>Yes, you can use <code>USE_NNPACK=0</code> to disable the NNPACK usage during a source build.</p>
<aside class="quote no-group" data-username="Zhicheng_Wang" data-post="1" data-topic="162146">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zhicheng_wang/48/53493_2.png" class="avatar"> Zhicheng_Wang:</div>
<blockquote>
<p>How will the running speed be influenced if I got rid of NNPACK?</p>
</blockquote>
</aside>
<p>I’m unsure what exactly the baseline would be, since your current approach isn’t working at all using NNPACK.</p>
Libtorch NNPACK

username: muhbdeir
cooked: <p>hi, I am new to pytorch and torchlib.<br>
I am trying to run a model on torchlib for cpp, the model was written and trained in python pytorch.<br>
I succeeded in reading the model in cpp but when I used the forward method I got the following error:<br>
[W NNPACK.cpp:51] Could not initialize NNPACK! Reason: Unsupported hardware.</p>
<p>I cannot figure out what is the problem here</p>

username: Zhicheng_Wang
cooked: <p>Some devs said it’s because NNPACK (a component of libtorch and pytorch) requires AVX2 instruction sets to initialize. See <a href="https://discuss.pytorch.org/t/bug-w-nnpack-cpp-80-could-not-initialize-nnpack-reason-unsupported-hardware/107518/5">Bug report about NNPACK</a></p>
<p>Are you running on a robot or some thing like that? I am facing the same issue on a robotic dog. It’s suggested to change the board CPU with a stronger one.</p>
Monotonic neural net in PyTorch

username: beardl
cooked: <p>I wanted to build a monotonic neural net in PyTorch, specifically such that the <strong>weights of all layers are restricted to be positive</strong>. I want to know if this is possible to do within the object class itself, instead of adding constraints during optimization or instantiation.</p>
<p>As an example, consider the network -</p>
<pre><code class="lang-auto">class NeuralNet(nn.Module):
    def __init__(self, in_features):
        self.layer1 = nn.Linear(in_features, 10, bias=False)
        self.layer2 = nn.Linear(10, 20, bias=False)

    def forward(self, x):
        return self.layer2(self.layer1(x))
</code></pre>
<p>Now, I would like to, at every forward pass, set every weight value which is negative (&lt;0) to 0. How do I do this appropriately? How do I ensure that autograd tracks weight updates correctly in this scenario?</p>

username: beardl
cooked: <p>Found a way!</p>
<pre><code class="lang-auto">class NeuralNet(nn.Module):
    def __init__(self, in_features):
        self.layer1 = nn.Linear(in_features, 10, bias=False)
        self.layer2 = nn.Linear(10, 20, bias=False)

    def forward(self, x):
        return self.layer2(self.layer1(x))

   def __call__(self):
        for module in self.modules:
            if hasattr(module, 'weight'):
                w = module.weight.data
                w = w.clamp(0.0, None)
                module.weight.data = w
</code></pre>
Difference between set parameter requires_grad=False and exclude them from optimizer?

username: x_u
cooked: <p>I want to freeze some layers (or sub model ), there are two methods:</p>
<ol>
<li>
<ul>
<li>set requires_grad =False</li>
</ul>
</li>
</ol>
<pre><code class="lang-auto">for v in  sub_model.parameters():
     v.requires_grad = False 
optimizer = torch.optim.Adam( self.model.parameters() )
</code></pre>
<ol start="2">
<li>
<ul>
<li>do not include these parameters to optimizer</li>
</ul>
</li>
</ol>
<pre><code class="lang-auto">optimizer = Adam([ v for k,v in self.model.named_parameters() if k not in self.sub_model.named_parameters() ] )
</code></pre>
<p>What is the difference and which should I use ?<br>
Thank you so much.</p>

username: ptrblck
cooked: <p>I would try to write the code as explicit as possible and filter the unneeded parameters out as described also e.g. <a href="https://discuss.pytorch.org/t/what-is-the-behavior-of-passing-model-parameters-with-requires-grad-false-to-an-optimizer/57817/4">here</a>. This would make sure that the optimizer does not have any references to these frozen parameters and will thus never update them.<br>
The first approach could also work, but be aware that the optimizer could still update frozen parameters if it has running stats (e.g. as is the case for <code>Adam</code>) for these parameters from previous iterations.</p>
Loss Function for predicting the probability distribution of labels

username: nassim
cooked: <p>Hi, i want to design a network for a prediction task. But my prediction task is quite special. I don’t want to train my model to predict labels but the probability of those labels (my targets are probabilities of those labels , not certainties). So my question is ,which loss function should i choose for this task? Do i still use cross_entropy or do i use something else ?</p>

username: KFrank
cooked: <p>Hi Nassim!</p>
<aside class="quote no-group" data-username="nassim" data-post="1" data-topic="162004" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nassim/48/53230_2.png" class="avatar"> nassim:</div>
<blockquote>
<p>Do i still use cross_entropy or do i use something else ?</p>
</blockquote>
</aside>
<p><code>CrossEntropyLoss</code> would be a perfectly appropriate – and likely the<br>
best – loss function for this use case.</p>
<p>(Older versions <code>CrossEntropyLoss</code> did not support probabilistic targets,<br>
but more recent versions, including the current stable release, 1.12.1, do.)</p>
<p>Assuming that <code>target</code> is a proper probability distribution, i.e.,<br>
<code>0.0 &lt;= target &lt;= 1.0</code> and <code>target.sum() == 1.0</code>,<br>
<code>CrossEntropyLoss() (pred, target)</code> will take on its minimum precisely<br>
when the probabilities derived from <code>pred</code>, <code>pred.softmax()</code>, are equal to<br>
<code>target</code>.  (Note that in general this minimum value will not be <code>0.0</code>.)</p>
<p>Let’s say that <code>target</code> has shape <code>[nBatch, nClass]</code>, that is, for<br>
each sample your <code>target</code> is a list of <code>nClass</code> label probabilities.<br>
You would want the final layer of your network to be a <code>Linear</code> with<br>
<code>out_features = nClass</code>.  Then feed the output of that <code>Linear</code><br>
directly to <code>CrossEntropyLoss</code> with no intervening <code>softmax()</code> nor<br>
other non-linearity.</p>
<p>Just to be clear, let’s say that <code>target = [0.75, 0.50, 0.25]</code>.  Your<br>
network will <em>not</em> try to learn to predict <code>[1.0, 0.0, 0.0]</code>.  Instead, it<br>
will try to learn to predict* precisely <code>[0.75, 0.50, 0.25]</code>.</p>
<p>*)  More correctly, your network will learn to predict unnormalized<br>
log-probabilities that yield <code>[0.75, 0.50, 0.25]</code> when passed through<br>
<code>softmax()</code>.</p>
<p>Best.</p>
<p>K. Frank</p>

username: nassim
cooked: <p>Hi KFrank  thanks for the answer!</p>
Self-made v.s. Pytorch-made: The processing difference

username: TAKAHITorch
cooked: <p>Hi,<br>
I’m new here. I wanna know the processing difference between self-made Deep Neural Network in Google Cola v.s. the one made of Pytorch.</p>
<p>I met a case where self-made one overflows while Pytorch-made one does not.</p>
<p>I mean, like, I build a DNN which predict where a ball gonna fall down once I give it parameters: Ball mass M, initial velocity and angle as V and THETA respectively.</p>
<p>When the parameter values are small(M=0-5, V=0-1), there is no problem. Both self-made and Pytorch-made work. However when the values are large (M=0-50, V=0-30), self-made one overflows. (Pytorch-made works.)</p>
<p>The errors are like ,</p>
<p>RuntimeWarning: overflow encountered in matmul</p>
<p>RuntimeWarning: invalid value encountered in matmul</p>
<p>Why could it happen? Is there some special processings in Pytorch? Or Does pytorch handle numbers in a different way?</p>

username: ptrblck
cooked: <p>Could you describe what “self-made” means in this context? Based on your description and the posted error messages it seems you are manually implementing a <code>matmul</code> operation which then overflows?</p>
Equivalent totorch_geometric.nn.norm.diff_group_norm in tensorflow

username: Kwather_Ali
cooked: <p>I need some explanation about this class and how to write that in TensorFlow can anyone help me, please</p>

username: ptrblck
cooked: <p>Since you are interested in a TF implementation, I would recommend to post this question in the TF discussion board as you would find more TF experts there.</p>
Get final feature and feature for every 'M' in VGG16

username: bryan123
cooked: <p>cfg = {<br>
‘VGG11’: [64, ‘M’, 128, ‘M’, 256, 256, ‘M’, 512, 512, ‘M’, 512, 512, ‘M’],<br>
‘VGG13’: [64, 64, ‘M’, 128, 128, ‘M’, 256, 256, ‘M’, 512, 512, ‘M’, 512, 512, ‘M’],<br>
‘VGG16’: [64, 64, ‘M’, 128, 128, ‘M’, 256, 256, 256, ‘M’, 512, 512, 512, ‘M’, 512, 512, 512, ‘M’],<br>
‘VGG19’: [64, 64, ‘M’, 128, 128, ‘M’, 256, 256, 256, 256, ‘M’, 512, 512, 512, 512, ‘M’, 512, 512, 512, 512, ‘M’],<br>
}</p>
<p>class VGG(nn.Module):</p>
<pre><code class="lang-auto">def __init__(self, vgg_name):
    super(VGG, self).__init__()
    self.cfg = cfg[vgg_name]
    self.teacher = self._make_layers()
    self.pool = nn.AvgPool2d(kernel_size=1, stride=1)
    self.linear = nn.Linear(512, 3) # change last layer to 3

def forward(self, x):
    teacher_counter = 0
    student_features = []
    feature = x
    for block in self.cfg:
        if block == 'M':
            feature = self.teacher[teacher_counter](feature)
            student_features.append(feature)
            teacher_counter += 1

    out = self.pool(feature)
    out = out.view(out.size(0), -1)
    out = self.linear(feature)

    print(len(student_features))
    return out, student_features


def _make_layers(self):
    layers = []
    teacher = []
    in_channels = 3
    for x in self.cfg:
        if x == 'M':
            teacher.append(nn.Sequential(*layers).to("cuda"))
            layers = []
        else:
            layers += [
                    nn.Conv2d(in_channels, x, kernel_size=3, padding=1),
                    nn.BatchNorm2d(x),
                    nn.ReLU(inplace=True)
                ]
            in_channels = x

    return teacher
</code></pre>
<p>The goal is to return the final feature of VGG16 and get features for each ‘M’ in student_features. Could anyone please let me know if it is the correct implementation?</p>
GPU being used for training alongside 100 % usage of CPU!

username: Reza_Marefat
cooked: <p>I have used the source code in <a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html" rel="noopener nofollow ugc">this</a>, which is an official source code in Pytorch website in a remote server for training a classifier. The device is cuda and during the training the GPU usage is verified. However, the usage of CPU jumps to more than 300% when I monitor it using htop. I have even tried to feed the model random tensors and the same happens (omits the possibility of DataLoader as a bottleneck). Any idea as a solution to keep the usage of CPU low would be appreciated. I use pip for pytorch (conda is not allowed in the server) and the version is the latest (1.12).</p>
What are the parameters involved in the gradient calculation of a GAN?

username: Jeet
cooked: <p>I have designed a GAN in which while training the discriminator I did -<br>
lossDiscriminator.backward(retain_graph=True).<br>
After that I do<br>
lossGenerator.backward()</p>
<p>I would like to know what are all parameters involved in calculating the gradient of the discriminator and the generator respectively. are all the parameters from both the neural networks are involved since I did retain_graph= True ?</p>
Shape of Conv1D followed by pooling layer

username: anorak_ransack
cooked: <p>I have a batch of variable length sequences. I pad them to a fixed length and then apply a <code>Conv1D</code> with <code>dilation</code> and  <code>strides &gt; 1</code> layer followed by <code>MaxPool</code> layer. I want to figure out the extent of the padding signal in the output sequence i.e. which part of the sequence is made up of contributions only from the padding tokens. Right now I am doing <code>output of conv layer = math.ceil(non_zero_seq_len / strides)</code> for both the convolution layer and the pooling layer. This gives an answer in the ballpark but I am not completely sure if its right. I would eventually like to mask these new tensors so I can pass these into a transformer later.</p>
<p>It would be great if i could just get some input on whether I am going about it in the right way.</p>
<p>Cheers</p>
Can anyone explain torch._assert with some examples , I am not able to understand from docs.Thanks in advance

username: Jayanth_Kumar
cooked: <p>Can anyone explain torch._assert with some examples , I am not able to understand from docs.Thanks in advance</p>

username: ptrblck
cooked: <p><code>torch._assert</code> should have the same functionality as the plain Python <code>assert</code> method, but can be used in e.g. <code>torch.fx</code> during tracing.<br>
You could stick to the default <code>assert</code> method unless you need to use it in some tracing methods.</p>
Model weight size and inference time for fp16 model using apex mixed precision with optimization level O3

username: SM19
cooked: <p>Hi,</p>
<p>I have successfully trained a model using Nvidia apex mixed precision plugin and saved weights as .pth<br>
I used optmization level O3. So I expect my model weights should be saved as FP16 weights.<br>
However, I see the size of .pth file is same as that of a FP32 trained model.</p>
<p>Could you please help me understand this?</p>
<p>Also, I haven’t timed the inference, but can I expect the inference to be faster than FP32?</p>
<p>My idea behind this is - I wish to use Tensorrt FP16, default path is train FP32, convert/ quantize to FP16 using torch-tensorrt. However, I see some performance degradation. Hence, may be if I train natively in FP16 and then do the conversion, it should give me better results.</p>
<p>Is this assumption sensible?</p>
<p>Please let me know.</p>
<p>Best Regards</p>

username: ptrblck
cooked: <p><code>apex.amp</code> is deprecated and the <code>O3</code> <code>opt_level</code> was used for “pure” FP16 training, which you could achieve by directly calling <code>.half()</code> on the model and input data without using <code>apex.amp</code>.</p>
Manual application of the chain rule

username: mmoelle1
cooked: <p>Dear all,</p>
<p>my loss function involves a term <code>f(x)</code> from which I want to compute part of the derivative by hand making use of the chain rule, i.e. <code> df(x)/dw = df(x)/dx * dx/dw</code>. The reason is that I have an explicit and pre-computable expression for <code>df(x)/dx</code>. The code is as follows</p>
<pre><code class="lang-auto">Tensor x;
Tensor f = compute_f(x);
Tensor df = compute_df_fx(x);
</code></pre>
<p>How can I make sure that the <code>df</code> term is used in the back propagation step?</p>

username: KFrank
cooked: <p>Hi Matthias!</p>
<aside class="quote no-group" data-username="mmoelle1" data-post="1" data-topic="162082" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mmoelle1/48/30239_2.png" class="avatar"> mmoelle1:</div>
<blockquote>
<pre><code class="lang-auto">Tensor f = compute_f(x);
Tensor df = compute_df_fx(x);
</code></pre>
<p>How can I make sure that the <code>df</code> term is used in the back propagation step?</p>
</blockquote>
</aside>
<p>Package <code>compute_f()</code> and <code>compute_df_fx()</code> as a custom <a href="https://pytorch.org/docs/1.12/autograd.html#function" rel="noopener nofollow ugc">autograd.Function</a>.<br>
Roughly speaking <code>compute_f()</code> will become your <code>Function</code>’s <code>forward()</code><br>
method, and <code>compute_df_fx()</code> will become <code>backward()</code>.</p>
<p>If it is more efficient or more convenient for your use case, you can compute<br>
<code>compute_df_fx()</code> in the forward pass (in <code>forward()</code>, store in <code>ctx</code> (the <em>context</em>),<br>
and then use it in the backward pass (in <code>backward()</code>).</p>
<p>Best.</p>
<p>K. Frank</p>

username: Matthias_Moller
cooked: <p>Hi K. Frank,</p>
<p>Thanks very much for your suggestion. Meanwhile, I also came across another solution <a href="https://discuss.pytorch.org/t/how-to-manually-do-chain-rule-backprop/2923/2">here</a> which seems easier. Do you know if <code>Variables</code> are supported in C++?</p>
How do I create GraphIE?

username: YuhskeHujisaki
cooked: <p>What is an overview of GraphIE’s architecture and the processing pipeline during learning and inference? What are the input and output data formats for each processing step in the pipeline?</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://arxiv.org/abs/1810.13083">
  <header class="source">
      <img src="https://static.arxiv.org/static/browse/0.3.4/images/icons/favicon.ico" class="site-icon" width="32" height="32">

      <a href="https://arxiv.org/abs/1810.13083" target="_blank" rel="noopener nofollow ugc">arXiv.org</a>
  </header>

  <article class="onebox-body">
    <img src="https://static.arxiv.org/icons/twitter/arxiv-logo-twitter-square.png" class="thumbnail onebox-avatar" width="500" height="500">

<h3><a href="https://arxiv.org/abs/1810.13083" target="_blank" rel="noopener nofollow ugc">GraphIE: A Graph-Based Framework for Information Extraction</a></h3>

  <p>Most modern Information Extraction (IE) systems are implemented as sequential
taggers and only model local dependencies. Non-local and non-sequential context
is, however, a valuable source of information to improve predictions. In this
paper, we...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Randomness in SubsetRandomSampler

username: sivannavis
cooked: <p>Hi! I have a question about the usage of <code>SubsetRandomSampler</code>.<br>
I have multiple dataloader in my code and say it’s defined this way:</p>
<pre><code class="lang-auto">gen = torch.Generator()
gen.manual_seed(seed)
data_loader = DataLoader(train_set,
        batch_size=int(args.batch_size),
        shuffle=False,
        sampler=torch_sampler.SubsetRandomSampler(range(100)),
        generator=gen)

eval_loader = DataLoader(eval_set,
                             batch_size=args.batch_size,
                             shuffle=False,
                             drop_last=False,
                             pin_memory=True)
</code></pre>
<p>The first loader is used before trainin and testing, the latter is used in testing. If I exclude testing in each epoch, with a fixed seed, the first dataloader gives the same batch order per epoch. But if I include testing in each epoch with a same seed, the first dataloader will give a different batch order.</p>
<p>It’s still the case even if I set the generator in <code>data_loader</code>. So what happened to the SubsetRandomSampler when I do testing right after it in every epoch?<br>
<a class="mention" href="/u/ptrblck">@ptrblck</a> sorry to bother you again, but could you please help me look at this?</p>

username: sivannavis
cooked: <p>In short, how can I make sure when I iterate <code>data_loader</code> with SubsetRandomSampler with the same seed, it’s giving a fixed order of batches per epoch?</p>
<p>Also in this case I don’t think SubsetRandomSampler and RandomSampler are different, so what’s the “subset” for?</p>
<pre><code class="lang-auto">from torch.utils.data import SequentialSampler, BatchSampler, SubsetRandomSampler, RandomSampler
list(BatchSampler(SubsetRandomSampler(range(10)), batch_size=3, drop_last=False))
Out[7]: [[2, 9, 5], [1, 0, 4], [8, 7, 3], [6]]
list(BatchSampler(SequentialSampler(range(10)), batch_size=3, drop_last=False))
Out[8]: [[0, 1, 2], [3, 4, 5], [6, 7, 8], [9]]
list(BatchSampler(RandomSampler(range(10)), batch_size=3, drop_last=False))
Out[9]: [[2, 6, 7], [9, 5, 4], [8, 1, 3], [0]]
</code></pre>