akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
NVIDIA A100 GPU - RuntimeError: cuDNN error: CUDNN_STATUS_MAPPING_ERROR

username: aroibu1
cooked: <p>Hi all - I started experiencing this issue recently on a new set of A100 GPUs. I am using the following configuration:</p>
<p>NVIDIA A100 GPU<br>
torch == 1.4.0<br>
Python == 3.7.4-GCCcore-8.3.0<br>
CUDA == 10.1.243-GCC-8.3.0<br>
cudnn == 7.6.5.32-10.1</p>
<p>The error is occurring when I am calling forward on a sequence of convolutional operations in my code.</p>
<p>I have used the same configuration on other GPUs, such as K80, P100, V100 and RTX8000, and never had the error. Anybody found a solution to this issue?</p>

username: aroibu1
cooked: <p>Just updated to the latest torch 1.8.1 and now the error has changed to:</p>
<pre><code class="lang-auto">....python3.7/site-packages/torch/cuda/__init__.py:104: UserWarning:
A100-PCIE-40GB with CUDA capability sm_80 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70.
If you want to use the A100-PCIE-40GB GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/
</code></pre>
<p>I had a look at the link provided, but I am not very sure how I can update or change my pytorch installation to allow sm_80.</p>

username: aroibu1
cooked: <p>I think I fixed my own problem, but posting the answer here in case anybody ever comes across this later down the line.</p>
<p>The first issue comes from the fact that (at least in my case) the A100 GPUs comes with sm_80, but also with pre-installed CUDA 11+. In this case, I was sabotaging myself by module loading CUDA 10.1 and cudnn 7.6.5.32 on the cluster.</p>
<p>The solution came by using the CUDA 11.1 version of pytorch from here: <a href="https://pytorch.org/get-started/locally/" class="inline-onebox" rel="noopener nofollow ugc">Start Locally | PyTorch</a></p>
<p><code>pip3 install torch==1.8.1+cu111 torchvision==0.9.1+cu111 torchaudio==0.8.1 -f https://download.pytorch.org/whl/torch_stable.html</code></p>
<p>Then, I just stopped loading CUDA 10.1 and cudnn 7.6.5 as separate modules, which made all errors go away. Running a few tests jobs now, and will post any updates in case other things get broken.</p>

username: OneSeer
cooked: <p>Thanks, Andrei–just wanted to let you know that this saved me today!</p>

username: akashs
cooked: <p>Did you build a public docker for this? If yes, could you please share it with us? Thanks.</p>

username: aroibu1
cooked: <p>Hi <a class="mention" href="/u/akashs">@akashs</a>. There is not need for this, as this is an installation solution. Just follow the steps in the solution.</p>
Transformer Training with shifted targets

username: maxrivera
cooked: <p>I was able to train my transformer down to a loss of less than 0.04 over 40 epochs, but I was not able to perform any sort of autoregressive inference. I realized that during training I was not shifting the target sentence to the right at all, which enables the model to predict the next word in the sequence during inference. So I shifted the target, but it completely ruined training. I can’t get a loss less than 4.0 over 40 epochs. Here I’ve attached my code and some outputs. If anyone could look at this I’d greatly appreciate it. My transformer model is based off of Harvard NLP’s modern PyTorch implementation.</p>
<pre><code class="lang-auto">optimizer=torch.optim.Adam(params=model.parameters(),lr=5e-4)
loss_fn = torch.nn.CrossEntropyLoss(ignore_index=0)
train_losses = []
val_losses = []
bleu_scores = []
for epoch in range(1,EPOCHS+1):

    # train loop 
    for i, (src,tgt) in enumerate(train_data):

        # place tensors to device 
        src = torch.Tensor(src).to(DEVICE).long()
        trg = torch.Tensor(tgt).to(DEVICE).long()

        tgt = trg[:,:-1] # decoder input (shifted left)
        tgt_y = trg[:,1:] # decoder target (shifted right)

        # print statements 
        print('tgt: ', id_to_word(tgt, en_index_dict))
        print('tgt_y: ', id_to_word(tgt_y, en_index_dict))
        print('tgt shape: ', tgt.size())
        print('tgt_y shape: ', tgt_y.size())

        src_mask = (src != PAD).unsqueeze(-2)
        tgt_mask_ = (tgt != PAD).unsqueeze(-2)
        tgt_mask = tgt_mask_ &amp; transformer_nlp.subsequent_mask(tgt.size(-1)).type_as(tgt_mask_.data)

        # forward pass 
        out = model.forward(src, tgt, src_mask, tgt_mask)

        # print prediction vs target sentence 
        val, ind = torch.max(out, -1)
        pred_sentence = id_to_word(ind,en_index_dict)
        print('out: ', id_to_word(ind, en_index_dict))
        

        # compute loss 
        train_loss = loss_fn(out.contiguous().view(-1, tgt_vocab), tgt_y.contiguous().view(-1))
        
        # backprop 
        optimizer.zero_grad()
        train_loss.backward()

        # update weights 
        optimizer.step()
</code></pre>
<pre><code class="lang-auto">src:  [['BOS', '你', '和', '我', '有', '相', '同', '的', '想', '法', '。', 'EOS', 'PAD',...]]
tgt:  [['BOS', 'you', 'and', 'i', 'have', 'the', 'same', 'idea', '.', 'EOS', 'PAD', 'PAD', 'PAD',...]]
tgt_y:  [['you', 'and', 'i', 'have', 'the', 'same', 'idea', '.', 'EOS', 'PAD', 'PAD','PAD','PAD','PAD'...]]
tgt shape:  torch.Size([64, 59])
tgt_y shape:  torch.Size([64, 59])
out:  [["'ve", 'cradle', 'artificial', 'minutes', 'letting', 'match', 'mysteries', ...]]
</code></pre>
Efficient summation over distance tensor blocks

username: cgoliver
cooked: <p>I have an N x N tensor, D (this is a symmetric distance matrix) the rows and columns of D are arranged in contiguous ‘groups’.  The grouping index for each row/col in D is given as a 1D tensor, B where B[i] is the group (int) assigned to row i of D. I would like to efficiently take the sum of all pairs of slices of D induced by the grouping B. For example, with a distance matrix between 5 objects in two groups:</p>
<pre><code class="lang-python">D = torch.tensor([[0., 1., 2., 3., 4.],
                  [1., 0., 1., 2., 3.],
                  [2., 1., 0., 1., 2.],
                  [3., 2., 1., 0., 1.],
                  [4., 3., 2., 1., 0.]])

# two groups, one of size 2 and one of size 3
B = torch.tensor([0, 0, 1, 1, 1])

# along each dim, the first group contains index 0 and 1
# and the second group has indices 2, 3, 4
# this makes 3 unique blocks: 
# D[0:2:,][:,0:2], D[0:2, :][:,2:5], D[2:5,:][:,2:5]

&gt;&gt;&gt; result
torch.tensor([2, 15, 8])
</code></pre>
<p>The most efficient would seem to be to make use of <code>scatter_add_</code> but would appreciate an example of how to apply it here, or other guidance.</p>

username: tom
cooked: <p>So scatter_add (and similar operations) when implemented with <code>atomicAdd</code> get slow when you have many “collisions”, i.e. GPU threads trying to write to the same address.<br>
When you have contiguous blocks like these, it is feasible to write a kernel that uses that to be much more efficient, but likely the effort is quite high.<br>
Personally, I’d probably double check what the performance is of  <code>scatter_add</code> vs. calling <code>.sum()</code> on the blocks.</p>
<p>Best regards</p>
<p>Thomas</p>

username: cgoliver
cooked: <p>Thanks for this, I’ll definitely do a benchmarking. I would also appreciate an example of how to use scatter_add in this situation since I am quite new to this function…</p>
NVIDIA-SMI 516.94 RuntimeError: No CUDA GPUs are available

username: naychelynn
cooked: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/5/757841432a6857a599668bc6a4740f59c6e411d5.png" alt="image" data-base62-sha1="gLbvyLui60Kz94ZrHlDDuMH3KPH" width="649" height="492"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/8/d/8d88da672c7c0d4c5c237e4aad0231e6e7eeec0e.png" alt="cuda" data-base62-sha1="kc4wQEwUFlIgtOEvGnL5vHovCt8" width="450" height="88"><br>
Hi, I’ve the following issue in my program. NVIDIA GeForce RTX 3080 with driver version 516.94 have been installed on my PC.<br>
I tried uninstall and reinstall cudatookit, pytorch, and reinstalling. torch.cuda.is_available() is still False though torch.version.cuda can check as<br>
11.3.<br>
Any guidance would be warmly welcome and thanks in advance.</p>

username: ptrblck
cooked: <p>Your system is most likely not able to communicate with the driver, which could happen e.g. if you didn’t restart the machine after a driver update.<br>
Make sure other CUDA samples are running first, then check PyTorch again.</p>

username: naychelynn
cooked: <p>Thanks for your suggestion. Now the CUDA is available but I’d like to know why CUDA device count 1 is displayed in red color?<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/2/d/2dba0c10f465764fd50592109dccf53e34468a15.png" alt="image" data-base62-sha1="6ww5BaN6vI5lfJ9mRq9al0pJQk5" width="331" height="195"></p>

username: ptrblck
cooked: <p>That’s your IDE’s color scheme and isn’t defined by PyTorch.</p>

username: coder-chica97
cooked: <p>Hey <a class="mention" href="/u/naychelynn">@naychelynn</a> , how did you solve the error? I am using GeForce RTX 3090.</p>

username: naychelynn
cooked: <p>I exactly follow <a class="mention" href="/u/ptrblck">@ptrblck</a> 's suggestion “Restart my PC”.<br>
I’ve tested on other environments too and it’s the same. ( installing torch without restarting then the same problem and CUDA is available after restart process).</p>
Load Pytorch Model Error in Python

username: joalar333
cooked: <p>I have this Issue when try to load Pytorch Model</p>
<p>this is the code:</p>
<h3>
<a name="load-model-1" class="anchor" href="#load-model-1"></a>LOAD MODEL</h3>
<p>device, epoch, model_name, model, optimizer_name,\</p>
<p>optimizer, optimizer_opts, loss_name, loss_func, weight_g,\</p>
<p>loss_history, loss_history_for_batch, valid_history,\</p>
<p>f2_score_history, epoch_loss_history, total_time_min = machine.load(</p>
<p>model_file, gpu_id</p>
<p>)</p>
<p>And show me this Error:</p>
<p>RuntimeError: Error(s) in loading state_dict for VGG11:<br>
size mismatch for linear_layers.6.weight: copying a param with shape torch.Size([3, 4096]) from checkpoint, the shape in current model is torch.Size([1, 4096]).<br>
size mismatch for linear_layers.6.bias: copying a param with shape torch.Size([3]) from checkpoint, the shape in current model is torch.Size([1]).</p>

username: ptrblck
cooked: <p>Based on the error message it seems <code>machine.load</code> initializes a model (VGG11) with a <code>linear_layers</code> <code>nn.Sequential</code> container and runs into a shape mismatch in layer6, which seems to be an <code>nn.Linear</code> module. I don’t know if this is the output layer, but if so I would guess your <code>state_dict</code> was stored from a model using a different number of classes than what is initialized in <code>machine.load</code>.</p>
Vmap causing TypeError with torchvision rotate

username: Art-MC
cooked: <p>I’m running into an issue trying to vmap over the torchvision rotate function. Rotate() requires an int or float input and does not accept single valued tensors. However vmap requires all batched inputs come as tensors. Additionally, one can’t create a helper function that calls rotate(im, angle.item()), as .item() calls are not allowed in vmap.</p>
<p>I feel like there should be a simple solution to this, but I’m not seeing it. Any help would be appreciated.</p>
<blockquote>
<p>from functorch import vmap<br>
from torchvision.transforms.functional import rotate<br>
vrot = vmap(rotate, in_dims=(0,0), out_dims=0)<br>
b, dim = 10, 64<br>
inp_ims = torch.rand((b, dim, dim))<br>
angles = torch.rand((b))<br>
vrot(inp_ims, angles)</p>
</blockquote>
<p>This gives: TypeError: Argument angle should be int or float</p>

username: ptrblck
cooked: <p>This might be a known limitation and I guess the inputs to <code>rotate</code> might need to be relaxed. Could you create an issue in the <a href="https://github.com/pytorch/vision/issues"><code>torchvision</code> repository</a> so that we can track and fix it?</p>

username: Art-MC
cooked: <p>I just opened an issue: <a href="https://github.com/pytorch/vision/issues/6524" class="inline-onebox" rel="noopener nofollow ugc">Vmap causing TypeError when applied to Rotate · Issue #6524 · pytorch/vision · GitHub</a><br>
Thanks!</p>
Adapting MNIST designed network for larger data

username: MarkM27
cooked: <p>I am attempting to implement <a href="https://github.com/vlukiyanov/pt-dec" rel="noopener nofollow ugc">this</a> deep clustering algorithm, which was designed to cluster the MNIST dataset. Single Channel 28x28 images.</p>
<p>The images I am trying to use are 416x416 and 3-Channel RGB. The script is initialised with the following functions.</p>
<pre><code class="lang-auto">class CachedMNIST(Dataset):
    def __init__(self, train, cuda, testing_mode=False):
        img_transform = transforms.Compose([transforms.Lambda(self._transformation)])
        # img_transform = transforms.Compose([transforms.Resize((28*28)), transforms.ToTensor(), transforms.Grayscale()])
        self.ds = torchvision.datasets.ImageFolder(root=train, transform=img_transform)
        self.cuda = cuda
        self.testing_mode = testing_mode
        self._cache = dict()

    @staticmethod
    def _transformation(img):
        return (torch.ByteTensor(torch.ByteStorage.from_buffer(img.tobytes())).float()
                * 0.02
        )
</code></pre>
<p>If the images are left un-altered the resulting tensor shape output from the <code>_transformation</code> function is of size <code>torch.Size{{256,519168]]</code> far too large for the AutoEncoder network to calculate.</p>
<p><strong>Error 1</strong><br>
<code>RuntimeError: mat1 and mat2 shapes cannot be multiplied (128x519168 and 784x500)</code></p>
<p>When I attempted to resize the images a the result is a 4D Tensor, <code>torch.Size([256,1,784,748])</code> which even when reducing the Batch Size to minuscule amounts the CUDA will crash as there is not enough memory.</p>
<p><strong>Error 2</strong><br>
<code>RuntimeError: CUDA out of memory.</code></p>
<p>I’m hoping someone can point me in the right direction to tackle this problem as there must be a more efficient way to adapt the network.</p>
<p><strong>AutoEnocder Model</strong></p>
<pre><code class="lang-auto">StackedDenoisingAutoEncoder(
  (encoder): Sequential(
    (0): Sequential(
      (linear): Linear(in_features=784, out_features=500, bias=True)
      (activation): ReLU()
    )
    (1): Sequential(
      (linear): Linear(in_features=500, out_features=500, bias=True)
      (activation): ReLU()
    )
    (2): Sequential(
      (linear): Linear(in_features=500, out_features=2000, bias=True)
      (activation): ReLU()
    )
    (3): Sequential(
      (linear): Linear(in_features=2000, out_features=10, bias=True)
    )
  )
  (decoder): Sequential(
    (0): Sequential(
      (linear): Linear(in_features=10, out_features=2000, bias=True)
      (activation): ReLU()
    )
    (1): Sequential(
      (linear): Linear(in_features=2000, out_features=500, bias=True)
      (activation): ReLU()
    )
    (2): Sequential(
      (linear): Linear(in_features=500, out_features=500, bias=True)
      (activation): ReLU()
    )
    (3): Sequential(
      (linear): Linear(in_features=500, out_features=784, bias=True)
    )
  )
)
</code></pre>
Autograd.grad with create_graph=True for Sparse Tensor

username: ago_yang
cooked: <p>I’m trying to use torch.autograd.grad with create_graph=True to calculate the gradient with respect to one parameter (Let’s say <strong>A</strong>) then calculate the gradient of the previous gradient with respect to other parameters (Let’s say <strong>B</strong>).</p>
<p>The thing is, <strong>A</strong> is dense and <strong>B</strong> is sparse. The following is an example.</p>
<pre><code class="lang-auto">x = torch.sparse_coo_tensor([[0], [1]], [1.], (2,2), requires_grad=True)
y = torch.tensor([[1., -1.], [1., 1.]], requires_grad=True)
n = torch.norm(torch.sparse.mm(x, y))
g = torch.autograd.grad(n, y, create_graph=True)
</code></pre>
<p>Then I get an error: " The backward pass for this operation requires the ‘self’ tensor to be strided, but a sparse tensor was given instead. Please either use a strided tensor or set requires_grad=False for ‘self’ "</p>
<p>I understand that torch.sparse.mm will produce SparseAddmmBackward grad_fn so it’s different from normal cases. But I can’t figure out a solution based on this fact.</p>
<p>I wonder if there is any way to solve this problem (To calculate the gradient of g with respect to x in the above code). Any suggestions will be greatly appreciated. <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=12" title=":grinning:" class="emoji" alt=":grinning:" loading="lazy" width="20" height="20"></p>

username: KFrank
cooked: <p>Hi Ago!</p>
<aside class="quote no-group" data-username="ago_yang" data-post="1" data-topic="159677" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ago_yang/48/52579_2.png" class="avatar"> ago_yang:</div>
<blockquote>
<p>The thing is, <strong>A</strong> is dense and <strong>B</strong> is sparse.</p>
</blockquote>
</aside>
<p>Perhaps this <a href="https://stackoverflow.com/q/73439223/" rel="noopener nofollow ugc">stackoverflow thread</a> will prove useful.</p>
<p>Best.</p>
<p>K. Frank</p>

username: ago_yang
cooked: <p>Hi Frank!</p>
<p>That thread is posted by me, but thanks for your reply.</p>

username: ago_yang
cooked: <p>I find <a href="https://github.com/rusty1s/pytorch_sparse#sparse-dense-matrix-multiplication" rel="noopener nofollow ugc">this function</a> can solve my problem.</p>
<pre><code class="lang-auto">x = torch.sparse_coo_tensor([[0], [1]], [1.], (2,2), requires_grad=True)
y = torch.tensor([[1., -1.], [1., 1.]], requires_grad=True)
x1 = x.coalesce()
n = torch.norm(torch_sparse.spmm(x1.indices(), x1.values(),
 x1.size()[0], x1.size()[1], y))
g = torch.norm(torch.autograd.grad(n, y, create_graph=True)[0])
g.backward()
</code></pre>
Apply ColorJitter transform to image with multiple channels

username: lorenzo_fabbri
cooked: <p>My images have 6 channels and I’d like to try adding <code>ColorJitter</code> to the set of transformations. Is there a way to use it with non-RGB images?</p>

username: vs74
cooked: <p>Any solution yet <a class="mention" href="/u/lorenzo_fabbri">@lorenzo_fabbri</a> ? I am interested in this as well.</p>
Training deeplabv3 with custom classifier

username: justin-larking-pk
cooked: <p>I attempting to create a deeplabv3 model with an effnet backbone. However I’m struggling with connecting a backbone not of the pre specified ones (i.e. torchvision.models.segmentation.deeplabv3_mobilenetv3_large etc)</p>
<p>I have:</p>
<pre><code class="lang-auto">def createdeeplabv3_effnet(outputchannels=1):
    """DeepLabv3 class with custom head
    Args:
        outputchannels (int, optional): The number of output channels
        in your dataset masks. Defaults to 1.
    Returns:
        model: Returns the DeepLabv3 model with the MobileNetV3 backbone.
    """
    backbone = torchvision.models.efficientnet_b0(pretrained=True, progress=True)
    model = torchvision.models.segmentation.DeepLabV3(backbone, None, None)
    #model = torchvision.models.segmentation.deeplabv3_resnet50(pretrained=True, progress=True)
    model.classifier = torchvision.models.segmentation.deeplabv3.DeepLabHead(1280, outputchannels)
    model.aux_classifier = torchvision.models.segmentation.deeplabv3.DeepLabHead(1280, outputchannels)
    # Set the model in training mode
    model.train()
    return model
</code></pre>
<p>This returns this error:<br>
lib/python3.9/site-packages/torchvision/models/segmentation/_utils.py", line 28, in forward<br>
x = features[“out”]<br>
IndexError: too many indices for tensor of dimension 2</p>
<p>While this:</p>
<pre><code class="lang-auto">def createdeeplabv3(outputchannels=1):
    """DeepLabv3 class with custom head
    Args:
        outputchannels (int, optional): The number of output channels
        in your dataset masks. Defaults to 1.
    Returns:
        model: Returns the DeepLabv3 model with the MobileNetV3 backbone.
    """
    model = torchvision.models.segmentation.deeplabv3_mobilenet_v3_large(pretrained=True, progress=True)
    #model = torchvision.models.segmentation.deeplabv3_resnet50(pretrained=True, progress=True)
    model.classifier = torchvision.models.segmentation.deeplabv3.DeepLabHead(960, outputchannels)
    # Set the model in training mode
    model.train()
    return model
</code></pre>
<p>Works perfectly fine.</p>
How can I classify the language of voice data?

username: YuhskeHujisaki
cooked: <p>How can I classify the language of voice data?  Specifically, I speak English, Japanese, French, Italian, and Russian to the voice data.  I want to make a model learn this and create a model that classifies what language the new voice data is speaking.  What kind of preprocessing, feature extraction, and model selection should be performed?</p>
Get the gradient of the network parameters

username: Wesker_Rongkai_Ma
cooked: <p>Guys, I am stucking on getting the gradients of a network’s parameters. So basically, what I wanna do is creating a long one-dimensional tensor which stores all the gradients of the parameters(weights, bias) of a network without updating the parameters with Optimizer_Network.step(), the reason why I have to use this one-dimensional tensor is that I have to do some other operations later with it. How can I get it?</p>

username: ptrblck
cooked: <p>You could iterate all parameters and store each gradient in a list:</p>
<pre><code class="lang-python">model = models.resnet50()
# Calculate dummy gradients
model(torch.randn(1, 3, 224, 224)).mean().backward()
grads = []
for param in model.parameters():
    grads.append(param.grad.view(-1))
grads = torch.cat(grads)
print(grads.shape)
&gt; torch.Size([25557032])
</code></pre>

username: Wesker_Rongkai_Ma
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thanks bro, It works!</p>

username: mahf
cooked: <p>Hi the solution of <a class="mention" href="/u/ptrblck">@ptrblck</a> works for me as well, but is there a more efficient way to do this? Possibly without a for loop especially for networks with large number of parameters.</p>

username: Megh_Bhalerao
cooked: <p>Yes, I have the same question. Is there any way to get the gradients of the parameters directly from the optimizer object without accessing the original parameters through the model object?<br>
Thanks!</p>

username: ptrblck
cooked: <p>Yes, you could use the stored references to the parameters and use them to check their gradients:</p>
<pre><code class="lang-python">for p in optimizer.param_groups[0]['params']:
    print(p.grad)
</code></pre>
<p>but I don’t know what kind of advantage it would give you.</p>

username: Megh_Bhalerao
cooked: <p>Thanks a lot. I think the advantage that I was looking for is that I can directly access the parameters which are to be optimized if they are a subset of the entire set of model parameters.</p>

username: ptrblck
cooked: <p>Ah OK, that makes sense. I was unsure about your use case as I thought you might be expecting a faster access or so.</p>
Can forward pass be broken into two (mutiple) steps

username: Krishna_Garg
cooked: <p>Can the full forward pass <code>f_x1</code> in PyTorch be done in 2 steps:</p>
<ul>
<li>Get <code>k</code>th hidden state representations <code>h_x1</code>
</li>
<li>Do forward pass on <code>h_x1</code> and get <code>12-k</code>th layer hidden state representations <code>f_x2</code>
</li>
</ul>
<p>I expected <code>f_x1</code> will be nearly equal to <code>f_x2</code>.</p>
<pre><code class="lang-auto">def forward(self, x_input_ids=None, x_seg_ids=None, x_atten_masks=None, inputs_embeds=None, k=0, get_hidden=False):
    
    if inputs_embeds is not None:
        # get hidden representations of (12-k)th layer
        outputs = self.bert(inputs_embeds=inputs_embeds, output_hidden_states=True)
        query   = outputs[2][12-k][:,0] # shape should be (batch_size, 768)
    
    else:
        # full forward pass
        outputs = self.bert(input_ids=x_input_ids, attention_mask=x_atten_masks, token_type_ids=x_seg_ids, 
                            output_hidden_states=True) # tuple of len 3
        query  = outputs[0][:,0]
        hidden = outputs[2] # returns a tuple of len 13. hidden[0] is embedding layer
    
    query   = self.dropout(query)
    linear  = self.relu(self.linear(query))
    out     = self.out(linear)
    if get_hidden:
        return out, hidden[k] # return kth hidden layer
    else:
        return out


criterion = nn.MSELoss(reduction='mean')
k = 3

# training
for i, batch in enumerate(train_loader):
    optimizer.zero_grad()
    sup_batch = [t.to(device) for t in batch]
    f_x1, h_x1 = model(*sup_batch[:3], get_hidden=True, k=k)
    f_x2 = model(inputs_embeds=h_x1, k=k)
    # Is f_x1 almost equal to f_x2 ??

    loss = criterion(f_x1, f_x2)
    loss.backward()
    nn.utils.clip_grad_norm_(model.parameters(), 1)
    optimizer.step()
</code></pre>
Adaptive top-k selection in machine learning

username: Taghreed
cooked: <p>In federated/ distributed learning, the server will send initially a global model to clients, and each client will train the model locally and then select the top k values, and send only these values to the server.</p>
<p><strong>How I can select an adaptive k in each client?</strong> rather than set top k value to fixed number (e.g. k=3 , which return top 3 values), I want to make the top k values adaptive, for example, some clients will send top 4 values, other may send 6 top values based on a defined feature ( largest value, largest loss , … etc)</p>
<p><strong>Is there any way to do that?</strong></p>
<p>Appreciate any help, thanks!</p>

username: tom
cooked: <p>You can code up any of these as you wish, but to not ruin efficiency, it is best to keep the number of points where the CPU gets tensors from the GPU to do control flow smallish.</p>
<p>Best regards</p>
<p>Thomas</p>

username: fduwjj
cooked: <p>I guess you want to ask how you can dynamically communicate num of results. I don’t think c10d collective supports that and you can consider using RPC for this. Let us know if this makes sense or not.</p>

username: Taghreed
cooked: <p>No, it is not what I mean.<br>
I mean: I want to reorder the large weights from clients in descending way, then taking the top large weights. let say some clients may send 3 top weights, other may send top 2 weights. so the top values will be dynamic not static. usually top k values is static (e.g., k = 3 , which takes only top 3 values), how to make it dynamic using a threshold ?<br>
How to do that? is there any algorithm or pseudo code to do that?</p>
Paging file problems when trying to import torch

username: desert_ranger
cooked: <p>Here’s my code -<br>
I was able to work with torch until last night. Now, suddenly I get this error -</p>
<pre><code class="lang-auto">
</code></pre>
<p>import numpy as np<br>
import gym<br>
import matplotlib.pyplot as plt<br>
import os<br>
import torch<br>
import random<br>
from torch import nn<br>
from torch.utils.data import DataLoader<br>
from torchvision import datasets, transforms<br>
from collections import deque<br>
import torchvision as tv<br>
import torch.nn.functional as F<br>
import sys<br>
env = gym.make(“ALE/Pong-v5”)</p>
<pre><code class="lang-auto">OSError                                   Traceback (most recent call last)
Input In [1], in &lt;cell line: 6&gt;()
      4 import matplotlib.pyplot as plt
      5 import os
----&gt; 6 import torch
      7 import random
      8 from torch import nn

File ~\anaconda3\envs\torch_1\lib\site-packages\torch\__init__.py:129, in &lt;module&gt;
    127     err = ctypes.WinError(last_error)
    128     err.strerror += f' Error loading "{dll}" or one of its dependencies.'
--&gt; 129     raise err
    130 elif res is not None:
    131     is_loaded = True

OSError: [WinError 1455] The paging file is too small for this operation to complete. Error loading "C:\Users\thoma\anaconda3\envs\torch_1\lib\site-packages\torch\lib\shm.dll" or one of its dependencies.
</code></pre>

username: ptrblck
cooked: <p>You might be running into <a href="https://discuss.pytorch.org/t/winerror-1455-the-paging-file-is-too-small-for-this-operation-to-complete/131233">thus issue</a>.</p>

username: desert_ranger
cooked: <p>Actually, there was something wrong with my Windows Virtual Memory settings. Thank you for your help though.</p>
Understanding vocab.txt of vinai/bertweet-base

username: Krishna_Garg
cooked: <p>While looking at vocab.txt <a href="https://huggingface.co/vinai/bertweet-base/raw/main/vocab.txt" rel="noopener nofollow ugc">here</a>, I was left wondering why the vocabulary for Bertweet is not continuous.<br>
For instance, see the below sample:</p>
<pre><code class="lang-auto">...
dice 63328
)@@ 63327
struggled 63326
wraps 63324
Investors 63312
#summer@@ 63305
...
</code></pre>
<p>As you can see, after 63305, we have 63312, followed by 63324… what about the numbers in between?</p>
<ol start="2">
<li>Also, it feels a bit strange why vocabulary starts at around 3800.</li>
</ol>
<pre><code class="lang-auto">르@@ 3800
utory 3798
...
</code></pre>
<p>Any explanations will be really appreciated.</p>
cannot assign 'torch.cuda.FloatTensor' as child module 'relation_embeddings' (torch.nn.Module or None expected)

username: jjyaoao
cooked: <p>That is so confusing to me! Can a friend help me?</p>

username: ptrblck
cooked: <p>Based on the error message it seems you are trying to replace a registered <code>nn.Module</code> with a tensor, which won’t work:</p>
<pre><code class="lang-python">class MyModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.relation_embeddings = nn.Embedding(10, 10)

model = MyModel()
model.relation_embeddings = torch.randn(1)
# TypeError: cannot assign 'torch.FloatTensor' as child module 'relation_embeddings' (torch.nn.Module or None expected)
</code></pre>
<p>Either replace it with another module or delete it by setting it to <code>None</code>.</p>
Understanding Vector-Jacobian Product

username: krypticmouse
cooked: <p>Hi,<br>
I’ve been reading about graphs and autograd and bit and came across the following:-</p>
<blockquote>
<p>Pytorch doesn’t compute Jacobian and instead uses VJP directly to calculate the derivates directly.</p>
</blockquote>
<p>So I had a few questions in mind:-</p>
<ol>
<li>What exactly does VJP do? I understand it gives us product between vector and Jacobian but what is that?</li>
<li>What is v, it’s cotangent but what role does it play? By default it’s a tensor of 1 which people keep saying is “differentiation by self”. But that’s just adding up the columns, how does adding columns give you derivative?</li>
<li>If possible, can someone explain how you calculate VJP directly from the graph?</li>
</ol>
<p>Been looking everywhere at this moment and with each article I read it seems no one directly addresses this topic. Kindly Help <img src="https://discuss.pytorch.org/images/emoji/apple/smiling_face_with_tear.png?v=12" title=":smiling_face_with_tear:" class="emoji" alt=":smiling_face_with_tear:" loading="lazy" width="20" height="20"></p>
<p>Thanks!</p>
How to prevent default gradient synchronization when calling loss.backward()?

username: NaomiEX
cooked: <p>Hi, I’m quite new to PyTorch but I’m using DistributedDataParallel (DDP) and I’m trying to implement gradient quantization for a project, i.e. I want to compress the gradients before synchronizing them.</p>
<p>From exploring around a bit, I know that <code>loss.backward()</code> automatically synchronizes the gradients. Is there a way to prevent this behaviour and to just get the local gradients then manually synchronize them myself?</p>

username: fduwjj
cooked: <p>You can use backward hook to add your customized logic here before the gradient is synced.</p>
Decoder only transformer model

username: Preludeskyline
cooked: <p>I am trying to run an ordinary differential equation within decoder only transformer model.<br>
My ultimate aim is to plot loss and training curves of the model upon reversing tokenization.<br>
However, I came across following error during training my model.<br>
Error:  <strong>IndexError</strong> : index out of range<br>
I could not have figured out what went wrong although tried to modify the code many times.<br>
Much appreciated any suggestion or help. Thanks.<br>
Below is the code:</p>
<p>import torch<br>
import torch.optim as optim<br>
from x_transformers import AutoregressiveWrapper, TransformerWrapper, Decoder</p>
<p>n_bins = 20<br>
n_tokens = n_bins * n_bins</p>
<p>def convert_to_tokens(Y, n_bins):<br>
digit_x = np.digitize(np.log(x), np.linspace(-3, 3, n_bins))<br>
digit_y = np.digitize(np.log(y), np.linspace(-3, 3, n_bins))<br>
return digit_y*n_bins + digit_y</p>
<p>model = TransformerWrapper(<br>
num_tokens = n_tokens,<br>
max_seq_len = 1024,<br>
attn_layers = Decoder(<br>
dim = 512,<br>
depth = 6,<br>
heads = 8<br>
)<br>
)</p>
<p>optimizer = optim.Adam(model.parameters(), lr=1e-3)</p>
<p>from scipy import integrate<br>
for epoch in range(20):</p>
<pre><code>optimizer.zero_grad
for i in range(100):
    a, b, c, d = 0.4, 0.002, 0.001, 0.7
    def f(Y, t):
        x, y = Y
        return [a * x - b * x * y,
        c * x * y - d * y]

    xy0 = [5, 4]
    t = np.linspace(0, 10, 250)
    Y = integrate.odeint(f, xy0, t)
    inputs = convert_to_tokens(Y, n_bins)
    inputs = torch.tensor(inputs).unsqueeze(0)

    logits = model(inputs)
    loss = torch.nn.functional.cross_entropy(logits, inputs)
    loss.backward()
    optimizer.step()
    
    running_loss += loss.item()
    if i % 5 == 0:print('[%d, %5d] loss: %.3f' % (epoch + 1, i + 1, running_loss / 20)) 
    running_loss = 0.0
</code></pre>
Why the DPRNN and DPGRU layers are not registered in the ModuleValidator

username: xichens
cooked: <p>I see the drop-in replacements DPRNN, DPGRU for the corresponding PyTorch modules are defined in the same file as the DPLSTM, but currently only the DPLSTM is registered in the <code>ModuleValidator</code>, not the other two. So <code>ModuleValidator.fix</code> does not attempt to replace the RNN or GRU in the model. Is there a particular reason for this?</p>

username: ffuuugor
cooked: <p>Good question - looks more like oversight on our side than anything else.</p>
<p>But with the recent <a href="https://github.com/pytorch/opacus/pull/445" rel="noopener nofollow ugc">changes</a> it’s likely we don’t need these drop-in replacements at all and will gradually remove them altogether</p>
Tune hyper-parameters for training a machine learning model

username: Aarya
cooked: <p>Training dataset : 588 images<br>
Validation dataset : 76 images</p>
<p>The config file looks like:</p>
<pre><code>&gt; "train_data_loader": {
</code></pre>
<blockquote>
<pre><code>    "type": "DataLoader",
    "args": {
        "batch_size": 1,
        "shuffle": true,
        "drop_last": true,
        "num_workers": 4,
        "pin_memory": true
    }
},
"val_data_loader": {
      "type": "DataLoader",
      "args": {
          "batch_size": 1,
          "shuffle": false,
          "drop_last": false,
          "num_workers": 4,
          "pin_memory": true
      }
  },

"optimizer": {
      "type": "Adam",
      "args": {
          "lr": 0.007,
          "weight_decay": 0.15,
          "amsgrad": true
      }
},
"lr_scheduler": {
    "type": "StepLR",
    "args": {
        "step_size": 30,
        "gamma": 0.1
    }
},

"trainer": {
    "epochs": 50,
    "gl_loss_lambda": 0.015,
    "crf_loss_lambda": 1,
    "cn_loss_lambda": 10000,
    "log_step_interval": 20,
    "val_step_interval": 9999,
</code></pre>
</blockquote>
<p>What could be the best set of parameters or a descent learning rate to start with?<br>
Currently i get poor results and due to early stopping it is not improved.<br>
Any suggestions ?</p>
How to input image sequences to a CNN+LSTM?

username: dllacer
cooked: <p>Hi everyone!</p>
<p>I have a neural network that starts with some convolutional layers, then an LSTM layer and finally some deconvolutional layers. Kind of encoder-decoder architecture with LSTM in the middle.</p>
<p>A sample in my dataset is a sequence of 4 images with shape <code>[4, 3, H, W]</code>. So, when I want to use batches, with <code>batch_size=8</code> for example, the resulting tensor would have shape <code>[8, 4, 3, H, W]</code>.</p>
<p>The problem is that Conv2d layer does not accept 5-dimensional tensors, it expects 4-dimensional tensors.</p>
<p>I’ve tried to reshape the tensor like this: <code>x.view(-1, 3, H, W)</code>, resulting this shape: <code>[32, 3, H, W]</code>. But this is not correct for me, as sequences are not related to each other, since they are picked randomly. So sequence 1 (4 contiguous images) in the batch is not the previous to sequence 2 (other 4 contiguous images) in this batch.</p>
<p>How can I feed my network with this 5-dimensional tensor in order to keep the batch information and also the sequence information?</p>

username: AladdinPerzon
cooked: <p>The CNN will only look at each image separate from the others, so it won’t look at them in a sequence in any way unless you would concatenate them in some way, like along the channels or something like that. It’s a little unclear what you’re trying to accomplish but from my understanding you want the CNN to do some feature extraction from the images and then you can use the lstm on those features.</p>
<p>So what I’m thinking you could do is what you said which is process each image separate by doing <code>x.view(batch_size*4, 3, H, W)</code> and then when each image has been processed by the CNN you will have shape <code>(batch_size*4, C, H_new, W_new)</code> and you can reshape this to the lstm by doing <code>x.view(batch_size, 4, -1)</code> if you’re using <code>batch_first=True</code> for the lstm. The lstm will then process them in a sequence.</p>

username: dllacer
cooked: <p>That’s totally true, it was just a reordering problem.<br>
Thank you so much for clarifying it for me.</p>

username: Bilel
cooked: <p>Can you tell me how you resolve it please, I really need it, how can we generate many labels from one sequence. Thank you in advance</p>

username: tejasri19
cooked: <p>Hey <a class="mention" href="/u/dllacer">@dllacer</a> , can you please share your code of the dataloader and model architecture for the model you used?</p>
How to get a new sparse tensor with grad

username: work
cooked: <p>Hi,<br>
I have a tensor <em>edge_v</em>(stores value) with grad and a 2d array <em>edge</em>(stores index), how can i get a sparse_coo_tensor with grad?</p>
<p>if i code like this, <em>sgraph</em> has no grad.<br>
sgraph =sp.coo_matrix((edge_v.tolist(), (edge[0], edge[1])),<br>
shape=(input.shape[0], input.shape[0]),<br>
dtype=np.float32)</p>
<p>if i code like this,it will get error.<br>
sgraph =sp.coo_matrix((edge_v, (edge[0], edge[1])),<br>
shape=(input.shape[0], input.shape[0]),<br>
dtype=np.float32)</p>
<p>this will get error either.<br>
sgraph = torch.sparse_coo_tensor(edge, edge_v, shape=(input.shape[0], input.shape[0]),dtype=np.float32)</p>

username: srishti-git1110
cooked: <p>Hi,<br>
The syntax is as follows :</p>
<pre><code class="lang-auto">torch.sparse_coo_tensor(indices, values, size=None, *, dtype=None, device=None, requires_grad=False)
</code></pre>
<p>Here <code>indices</code> represent the coordinates of the non-zero values in the matrix, and thus should be 2-dimensional. <code>indices = edge</code> in your case.</p>
<p>So, try this-<br>
<code>sgraph = torch.sparse_coo_tensor(indices=edge, values=edge_v, dtype=torch.float32, requires_grad=True)</code></p>
<p>Additionally, size argument is inferred automatically in case you do not manually provide a size for the sparse tensor required.</p>
<p>Hope this helps,<br>
Srishti</p>

username: work
cooked: <p>Thanks for your solution, it works!</p>
<blockquote>
<p>sgraph = torch.sparse_coo_tensor(indices=edge,<br>
values=edge_v,dtype=torch.float32,requires_grad=True)</p>
</blockquote>
How to use pretrained BERT in News Recommend

username: Nguyen_Duy
cooked: <p>Hello everyone.<br>
I want to use the BERT pre-training model for the news recommendation problem. In this problem, a news item has a lot of information such as title, summary, and content.<br>
Should I use a single BERT model for all 3 of the above fields of the news, or declare 3 separate BERT models (of course the same) for each of the above fields?<br>
Thank you.</p>

username: tom
cooked: <p>It probably is more efficient and just as well to use the same.</p>
<p>Best regards</p>
<p>Thomas</p>
Why pytorch behave differently while doing floating point division with different types of divisors

username: Marxlp
cooked: <pre><code class="lang-python">a1 = torch.tensor(858.8350830078125, dtype=torch.float32).cuda()/100.0

a2 = torch.tensor(858.8350830078125, dtype=torch.float32).cuda()/torch.tensor(100.0).cuda()

print(a1 - a2)
# out: tensor(-9.5367e-07, device='cuda:0')
</code></pre>
<p>python: 3.8.10<br>
pytorch: 1.8.1+cu111</p>

username: tom
cooked: <p>The difference of ~1e-6 is within the numerical accuracy of floating point operations that are of the result size, so this magnitude of difference is “normal operations”.<br>
Dividing by a scalar will use a different kernel than dividing by a cuda tensor. This is speculation, but one difference between the two is that the first will have the 100 passed around as fp64 before moving to fp32 on the GPU, maybe that makes some rounding difference.</p>
<p>Best regards</p>
<p>Thomas</p>
How to do transfer learning by freezing certain layers on different architectures

username: bing
cooked: <p>Hi,<br>
I am trying to do binary classification using transfer learning.<br>
In the process, I want to experiment with freezing/unfreezing different layers of different architectures but so far, I am able to freeze/unfreeze entire models only.<br>
Can anyone help me in illustrating it with a couple of model architectures?<br>
Below, I am using <code>Timm</code> and a couple of architectures - <code>convnext</code> and <code>resnet</code> -</p>
<pre><code class="lang-auto">import timm
convnext = timm.create_model('convnext_tiny_in22k', pretrained=True,num_classes=2) # noqa
resnet = timm.create_model('resnet50d', pretrained=True,num_classes=2)

</code></pre>
<p>How to find which layers to freeze/unfreeze?<br>
Thanks.</p>

username: tom
cooked: <p>Typically, the last few layers are unfrozen (e.g. just the “head”) and the remainder remains frozen.</p>
<p>Best regards</p>
<p>Thomas</p>