akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
RTX 2060 CUDA or TensorCores dont work

BinarySoul: <p>Hi,</p>
<p>I am very new to the topic. I was following a tutorial about simple AI and such. Nothing worked and the reason seems to be, that Python cannot Communicate with CUDA (Or my Graphicscard in general.).</p>
<p>So I learned that every Pytorch comes with its own CUDA Version. OFC I installed the newest, which probatly was the problem itself. My Graphic Card has CUDA 7.5 and 240 Tensor Cores (Gen 2).<br>
According to <a href="https://pytorch.org/get-started/previous-versions/" rel="noopener nofollow ugc">this</a> list, I need PyTorch 0.4.1 which was a real hustle to unistall the new one and install the old version.<br>
Now my Tutorial is not working anymore, because it was not meant to process with a historic version of Pytorch.</p>
<p>Does that mean I have no chance to work with AI (of this century) at all? Or do you guys know a trick to surepass the limitation?</p>
<p>GPU: INNO3D GeForce RTX 2060 Compact X1<br>
CPU: Intel Core I7 4790k<br>
RAM: 16 GB DDR3<br>
OS: Windows 10<br>
Python: 3.10.4<br>
All drivers up to date, CUDA Toolkit installed, Path Variables checked and seemed ok.</p>

ptrblck: <aside class="quote no-group" data-username="BinarySoul" data-post="1" data-topic="187441">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/e9c0ed/48.png" class="avatar"> BinarySoul:</div>
<blockquote>
<p>My Graphic Card has CUDA 7.5</p>
</blockquote>
</aside>
<p>“7.5” refers to the compute capability of your GPU not the CUDA toolkit release.<br>
Your device is supported in the current stable and nightly releases as the binaries support all devices between compute capabilities 3.7 to 8.9 using the PyTorch binaries shipping with CUDA 11.7 and up to 9.0 using the PyTorch binaries shipping with CUDA 11.8 and 12.1.</p>
Pytorch DDP with torchrun and slurm invalid device ordinal error

shrutishrestha: <p>I am running PYTORCH DDP (multinode) with horovod, slurm. Here when I use</p>
<ol>
<li>2 nodes, 1-1 gpu each it works perfectly</li>
</ol>
<p>but when I try</p>
<ol start="2">
<li>
<p>2 nodes, 2-2 gpu each<br>
each process only sees<br>
Number of available CUDA devices: 1</p>
<p>and when I do:<br>
torch.cuda.set_device(int(os.environ[“LOCAL_RANK”]))<br>
First GPU takes cuda 0, works fine but another GPU in same node throws</p>
<p>packages/torch/cuda/<strong>init</strong>.py", line 350, in set_device<br>
torch._C._cuda_setDevice(device)<br>
RuntimeError: CUDA error: invalid device ordinal</p>
</li>
</ol>
<p>I am guessing, slurn internally chooses 1 gpu at random, and we are providing something else in the setdevice so device id doesnot match</p>
<ol>
<li>how can we solved this issue?<br>
2: how can we get available device id in each machine in ddp? (i am only able to get name of GPUS, but that doesnot help as all GPU have same names)</li>
</ol>
CUDA device error: CUDA-capable device is/are busy

Zhaoyu_Wang: <p>Has anyone met this problem before? I ordered 2 GPU on one node<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/b/eb97b7127c0c658155d94d802cfb738872ab0ffd.png" data-download-href="https://discuss.pytorch.org/uploads/default/eb97b7127c0c658155d94d802cfb738872ab0ffd" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/b/eb97b7127c0c658155d94d802cfb738872ab0ffd.png" alt="image" data-base62-sha1="xC9fJh1dBdRYibKBmvz24cpmWId" width="690" height="229" data-dominant-color="1D1D1D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1894×631 64.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>What does “I ordered 2 GPUs” mean? Are you using a shared node? If so, is another user blocking the devices? Are you able to use both devices standalone?</p>

Zhaoyu_Wang: <p>Hi, they are on a shared node.<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/1/01684be345415085b60c7e8be0fbe1a574255023.png" data-download-href="https://discuss.pytorch.org/uploads/default/01684be345415085b60c7e8be0fbe1a574255023" title="1691659566868.png"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/1/01684be345415085b60c7e8be0fbe1a574255023.png" data-base62-sha1="crVKTIVj9hQiapfemOiuqNDROr" alt="1691659566868.png" width="388" height="500" data-dominant-color="1E1E1D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1691659566868.png</span><span class="informations">652×838 28.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>In that case the error is expected since all your GPUs are in <code>Exclusive Process</code> compute mode.</p>

Zhaoyu_Wang: <p>Sorry for the late reply and thank you so much for your suggestion!</p>
In Pytorch, what is stored for backward and why?

npz7yyk: <p>I am building a machine learning framework and need to understand what is stored for backward. Therefore, I checked out torch.autograd.graph.saved_tensors_hooks for some insights. Here is the code that I ran:</p>
<pre><code class="lang-auto">import torch.nn as nn
import torch


class MyModule(nn.Module):
    def __init__(self, module, name=""):
        super().__init__()
        self.module = module
        self.name = name

    def forward(self, *args, **kwargs):
        def pack_hook(tensors):
            print("in forward hook of", self.name, tensors.shape)
            return tensors

        def unpack_hook(tensors):
            print("in backward hook of", self.name, tensors.shape)
            return tensors

        with torch.autograd.graph.saved_tensors_hooks(pack_hook, unpack_hook):
            rst = self.module(*args, **kwargs)
            return rst


net = nn.Sequential(
    MyModule(nn.Linear(3, 5), "m1"),
    MyModule(nn.Linear(5, 7), "m2"),
    MyModule(nn.Linear(7, 9), "m3")
)


x = torch.randn(2, 3)
x = net(x)
</code></pre>
<p>And here is the result:</p>
<pre><code class="lang-auto">in forward hook of m1 torch.Size([2, 3])
in forward hook of m2 torch.Size([5, 7])
in forward hook of m2 torch.Size([2, 5])
in forward hook of m3 torch.Size([7, 9])
in forward hook of m3 torch.Size([2, 7])
</code></pre>
<p>After checking out tensors with shape [5, 7] and [7, 9], I found that they were actually transposed version of each linear weight respectively.<br>
I am confused, why the weight of m1, which has shape [3, 5], was not captured by saved_tensors_hooks?<br>
Many thanks!</p>

ptrblck: <p>The first <code>weight</code> was not stored since the <code>x</code> input does not require gradients so there is no need to store the <code>weight</code> of the first linear layer for the <code>dgrad</code> calculation.<br>
Use <code>x = torch.randn(2, 3, requires_grad=True)</code> and the <code>weight</code> of the first layer will be stored as well.</p>
Incomplete/sparse Gemm (for some sort of local attention) with only given indices

vadimkantorov: <p>I would like to do an incomplete Gemm, where we compute the dot products only with a fixed number of other other matrix rows (as computing the full matrix would not fit in memory).</p>
<p>Given two tensors: <code>emb[B, T, C]</code> and <code>ind[B, T, K]</code> where <code>ind[b, t, :]</code> contains indices of the neighborhood of <code>(b, t)</code>.</p>
<p>I would like to compute <code>out[b, t, k] = \sum_c emb[b, t, c] * emb[b, ind[b, t, k], c]</code>.</p>
<p>Can I achieve this in PyTorch in a vectorized way without a Python loop over <code>b</code> and <code>t</code> and without materializing intermediate tensors?</p>
<p>This would enable a lot of experimentation in sparser attention patterns</p>

vadimkantorov: <p>Maybe the question can be formulated also as: does PyTorch support dense gemm with a sparse output mask? (so that the output is sparse and only computed where the mask is non-zero)</p>
<p>Is <a href="https://pytorch.org/docs/stable/generated/torch.sparse.sampled_addmm.html" class="inline-onebox" rel="noopener nofollow ugc">torch.sparse.sampled_addmm — PyTorch 2.0 documentation</a> doing this?</p>
<p>Hmm, seems so, but does not support batches <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"> <a href="https://github.com/pytorch/pytorch/issues/105319" class="inline-onebox" rel="noopener nofollow ugc">Implementation of torch.sparse.sampled_baddmm · Issue #105319 · pytorch/pytorch · GitHub</a></p>
<p><a class="mention" href="/u/alband">@albanD</a></p>
Production of LSTM example

Matteo_Panfilo: <p>Hi everyone,</p>
<p>I am trying to code a very simple LSTM, below how I defined the main class:</p>
<pre><code class="lang-auto">class lstm_mdl(nn.Module):
    def __init__(self, x, n_nrns, nl, y):
        super(lstm_mdl, self).__init__()
        self.lstm = nn.LSTM(input_size = x, hidden_size = n_nrns, num_layers = nl, batch_first=True)
        self.linear = nn.Linear(n_nrns, y)
    def forward(self, x):
        y, _ = self.lstm(x)
        x = self.linear(y[:, -1, :])
        return x
</code></pre>
<p>Now, assuming three time-series y, A and B: we want to predict y with the time-series A and B. The time-series have 10 observations and we want a lookback of 1 period (2 periods per-batch).<br>
In doing this, we run the following code on the two tensors y and (A + B), to get the 3D tensors we use in order to train and compute deviations:</p>
<pre><code class="lang-auto">x = tc.tensor(A_B, dtype = tc.float32) #torch.Size([10, 2])
y = tc.tensor(y.reshape(-1, 1), dtype = tc.float32) #torch.Size([10, 1])

data_size = x.shape[0] - lookback

x_train = tc.zeros(data_size, lookback, x.shape[1])
y_train = tc.zeros(data_size, 1)
for i in range(data_size):
    x_train[i] = x[i:(i + lookback)] #torch.Size([8, 2, 2]) as the loop ends
    y_train[i] = y[i + lookback] #torch.Size([8, 1]) as the loop ends
</code></pre>
<p>This is going to provide use the usual 3D input-tensor, whose each batch represents observations in the range [t - (t-lookback)], [t-1, (t-loockback-1)] … and so far and so on. Thus, we get the usual sequences that contain moving windows extrapolated from the original dataset.</p>
<p>Finally, this is the loop for training:</p>
<pre><code class="lang-auto">for epoch in range(n_epochs):   
    epoch_loss = []
    for i in range(0, data_size, batch_size):
       
        input = x_train[i:(i + batch_size)]
        output = y_train[i:(i + batch_size)]
        pred = ann(input)
        current_loss = loss(pred, output) 
        optimizer.zero_grad()
        current_loss.backward()
        optimizer.step()
        epoch_loss.append(np.sqrt(current_loss.item()))
   rec_losses.append(np.array(epoch_loss).mean())
</code></pre>
<p>The loss decreases, but when I try tu run this I get weird results:<br>
<code>f_pred = ann(x_train)</code> <span class="hashtag-raw">#torch</span>.Size([8, 2, 2])</p>
<p>I would expect a prediction for each batch, so 8 different forecasts in total.<br>
For each batch the forecast is exactly the same instead (I can provide numbers if needed).</p>
<p>Can you help with my code?</p>
Training NN for multi-label data (binary bit array of dim 50)

nil_tdr: <p>Hello!</p>
<p>This is my problem setup.<br>
Train Input size (6300x300) These are standard BERT embeddings, so floating point numbers, mostly negatives.<br>
Train Output size (6300x50) These are binary bit arrays like [0, 0, 1, 1, 0, … 0]<br>
I am using a validation dataset of size 800.</p>
<p>I want to learn a NN network (with two hidden layers) that can map between input to output of train data. I have tried BCEloss. i played with learning rate, weight decay, dropout probability, batch size-these parameters. I also increased hidden parameter size to 1000, changed number of layers. But apparently my validation loss does not decrease or it increases. Training loss stops decreasing after some epochs.</p>
<p>I think my challenge is I have too many labels. I am careful to increase hidden layer size too much as that will increase NN complexity. Eventually i will try to scale my model to 100k sample size.</p>
<p>Can you please help me how I can design NN for this problem.</p>
<p>Thanks,<br>
Nil</p>
There is one more process during DDP training

alex18812649207: <p>I’m trying to DDP train a MNIST image classification, the code is as follows.</p>
<pre data-code-wrap="python"><code class="lang-python">def main():
    dist.init_process_group("nccl")
    rank = dist.get_rank()
    print(f"Start running DDP on rank {rank}.")
    
    transform = transforms.Compose([transforms.Resize((32, 32)), transforms.ToTensor()])
    train_dataset = MNIST("./MNIST", train=True, download=True, transform=transform)    # 训练数据集
    train_sampler = DistributedSampler(train_dataset, shuffle=True)
    train_dataloader = torch.utils.data.DataLoader(train_dataset, batch_size=256, sampler=train_sampler)

    device_id = rank % torch.cuda.device_count()
    model = LeNet().to(device_id)
    model = DDP(model, device_ids=[device_id])
    
    epochs = 10       # 定义训练轮数
    criterion = nn.CrossEntropyLoss()    # 定义交叉熵损失函数
    # 定义随机梯度下降优化器
    optimizer = optim.SGD(model.parameters(), lr=0.01, momentum=0.9, weight_decay=5e-4)

    for epoch in range(epochs):
        train_loss, correct, total = 0, 0, 0
        for index, (inputs, targets) in enumerate(train_dataloader):
            # 将输入数据放置在正确的设备上
            inputs, targets = inputs.to(device_id), targets.to(device_id)
            optimizer.zero_grad()
            outputs = model(inputs)               # 通过模型输出
            loss = criterion(outputs, targets)    # 计算每个 batch 的损失
            loss.backward()                       # 损失函数反向传播
            optimizer.step()                      # 随机梯度优化器对模型参数优化

            train_loss += loss.item()
            _, predict = outputs.max(1)
            total += targets.size(0)
            correct += predict.eq(targets).sum().item()
        print(f"epoch on rank {rank}: {epoch + 1} / {epochs}, loss: {loss:.4f}, accuracy: {(100 * correct / total):.2f}%")
        
    if rank == 0:  # 只在一个进程中保存模型，以防止多个进程尝试写入同一个文件
        torch.save(ddp_model.module.state_dict(), 'model.pth')    # 保存模型参数
        
    dist.destroy_process_group()

if __name__ == "__main__":
main()
</code></pre>
<p>Then I ran the code via torchrun.</p>
<pre data-code-wrap="bash"><code class="lang-bash">
torchrun --nnodes=1 --nproc_per_node=2 main.py

</code></pre>
<p>Then I used <code>watch -n 1 nvidia-smi</code> to check the process and found that there were three processes, two of which were on GPU 0.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/c/4c24024fefd61897bde3b7d74a679d85b22266af.jpeg" alt="image" data-base62-sha1="aRzr4RJbsx413gH1E2zife5nAmP" width="553" height="248"></p>
<p>Why 3 processes instead of 2?</p>
Latest conda seems incompatible with the brew cmake ninja

davidradl: <p>I am getting error in my development build. So I followed <a href="https://github.com/pytorch/pytorch#from-source" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch#from-source</a>:</p>
<pre><code class="lang-auto"> brew install cmake ninja
</code></pre>
<p>I get this error:</p>
<pre><code class="lang-auto">The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    conda-23.7.3               |  py310hbe9552e_0        1009 KB  conda-forge
    ------------------------------------------------------------
                                           Total:        1009 KB

The following packages will be UPDATED:

  conda                              23.5.0-py310hbe9552e_1 --&gt; 23.7.3-py310hbe9552e_0 

The following packages will be DOWNGRADED:

  cryptography                       41.0.1-py310hdd3b5e7_0 --&gt; 38.0.4-py310hfc83b78_0 


Proceed ([y]/n)? n
</code></pre>
<p>Following the PyTorch setup instructions looks like the latest conda is incompatible with the brew cmake ninja.</p>

davidradl: <p>I have pressed y but am now not sure how sane my environment is.</p>
Source build failing with ARMIE_COMMAND ARMIE_COMMAND-NOTFOUND

davidradl: <p>My source build was working, prior to my vacation (around 5 weeks ago). I am on an M1 mac</p>
<p>I notice that that the mac build instructions have changed. I have brew installed llvm. I run the setup command that is in the latest docs.</p>
<pre><code class="lang-auto">export CMAKE_PREFIX_PATH=${CONDA_PREFIX:-"$(dirname $(which conda))/../"}

MACOSX_DEPLOYMENT_TARGET=10.9 CC=clang CXX=clang++ python setup.py build --cmake-only
ccmake build
</code></pre>
<pre><code class="lang-auto">
I get an output :slight_smile: ARMIE_COMMAND                    ARMIE_COMMAND-NOTFOUND                       
 ATEN_INSTALL_BIN_SUBDIR          bin                                          
 ATEN_INSTALL_INCLUDE_SUBDIR      include                                      
 ATEN_INSTALL_LIB_SUBDIR          lib                                          
 ATEN_NO_TEST                     OFF                                          
 ATEN_THREADING                   OMP                                          
 AT_INSTALL_BIN_DIR               bin                                          
 AT_INSTALL_INCLUDE_DIR           include                                      
 AT_INSTALL_LIB_DIR               lib                                          
 AT_INSTALL_SHARE_DIR             share                                        
 BENCHMARK_BUILD_32_BITS          OFF                                          
 BENCHMARK_DOWNLOAD_DEPENDENCIE   OFF                                          
 BENCHMARK_ENABLE_ASSEMBLY_TEST   OFF                                          
 BENCHMARK_ENABLE_DOXYGEN         OFF                                          
 BENCHMARK_ENABLE_EXCEPTIONS      ON                                           
 BENCHMARK_ENABLE_GTEST_TESTS     ON                                           
 BENCHMARK_ENABLE_INSTALL         OFF                                          

ARMIE_COMMAND: Path to a program.                                               
Keys: [enter] Edit an entry [d] Delete an entry             CMake Version 3.26.4
      [l] Show log output   [c] Configure
      [h] Help              [q] Quit without generating
      [t] Toggle advanced mode (currently off)
</code></pre>
<p>I think the issue os that cmakecache.txt contains line<br>
ARMIE_COMMAND:FILEPATH=ARMIE_COMMAND-NOTFOUND<br>
This file is generated from the cmake command - so I cannot just comment this line out.<br>
I do not have an armclang++ to run here - I am running clang++.</p>
<p>any thoughts?</p>

davidradl: <p>I force reinstalled conda and this went away.</p>
Is torchtext still a live project?

iyzg: <p>I had been trying to go through the basic sentiment classification on the docs <a href="https://pytorch.org/tutorials/beginner/text_sentiment_ngrams_tutorial.html" rel="noopener nofollow ugc">here</a>, but I got an error of ‘NoneType’ object has no attribute ‘Lock’. This was an open issue on Github from a few months ago, but there hasn’t been any commits or real updates since then.</p>
<p>Does anyone know of alternative ways to load text that aren’t with torchtext since it doesn’t seem to be functioning anymore?</p>

vdw: <p>I think they made quite some changes between version; see also this <a href="https://discuss.pytorch.org/t/cannot-import-name-field-from-torchtext-data/166885">post</a></p>
<p>Maybe many tutorials won’t work out of the box anymore.</p>
How to change model output shape, so that model produces number of outputs = number of classes, without much effecting the accuracy?

palguna_gopireddy: <p>I have trained a dataset having 5 different classes, with a model that produces output shape <code>[Batch_Size, 400]</code> using <a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html#torch.nn.CrossEntropyLoss" rel="noopener nofollow ugc">Cross Entropy Loss</a> and Adam Optimizer<a href="https://pytorch.org/docs/stable/generated/torch.optim.Adam.html" rel="noopener nofollow ugc">Adam Optimizer</a> without <a href="https://pytorch.org/docs/stable/generated/torch.nn.Softmax.html#torch.nn.Softmax" rel="noopener nofollow ugc">Softmax Function</a>.</p>
<p>After I get the predicted outputs <code>y_predict = model(images.to(device)) </code> which is of shape <code>[Batch_Size, 400]</code> . I used <code>_, pred_labels = torch.max(y_predict, 1)</code> to get the predicted labels, which is of shape <code>[Batch_Size]</code> . Then I compared it with actual labels of that batch <code>true_labels</code> using <code>running_corrects += torch.sum(pred_labels == true_labels.data.to(device))</code> to get the number of corrects.</p>
<p>In the model output with size <code>[Batch_Size, 400]</code> each row with column indices 0 to 4 has non-zero values (My number of classes is 5). In each row from 5 to 400 all the values are zero. So When I used <code>_, pred_labels = torch.max(y_predict, 1)</code> , the predicted_labels are coming from 1st 0 to 4 column indices. My model produced 97% accuracy.</p>
<p>Is there anyway I can make:<br>
<code>torch.max</code> function only uses 0 to 4 column indices of each row to get the corresponding label?<br>
(Or)<br>
Getting <code>[Batch_Size, 5]</code> instead of <code>[Batch_Size, 400]</code> without much effecting the accuracy.</p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/palguna_gopireddy">@palguna_gopireddy</a>,</p>
<p>You can just use slicing to get the first 5 columns,</p>
<pre><code class="lang-auto">x = torch.randn(batch_size, 400) #random output data
x_5col = x[:,:5] #get first 5 columns
x_5col.shape #returns [batch_size, 5]
</code></pre>

palguna_gopireddy: <p>Thank you. Is this valid as correct way of training, if I put <code>y_predict[:,:5]</code> in the training after <code>y_predict = model(images.to(device))</code> in the training stage? (or) putting in the model itself.</p>

AlphaBetaGamma96: <p>That will depend on how your model is defined, and you may need to make changes accordingly.</p>
Set diagonal of each matrix in a batch to 0

Usman_Mahmood: <p>I have a tensor of shape N * M * M, where N is the batch size. I want to set the diagonal of each M*M matrix to a specific number (e.g. 0). I tried fill_diagonal_ method but it doesn’t work if N!=M.</p>
<p>I can loop through the batch dimension, but doesn’t want to create a loop.</p>
<p>Help would be appreciated.</p>

Eta_C: <pre><code class="lang-python">N, M, value = 5, 3, 3.1415926
x = torch.rand(N, M, M)
mask = torch.eye(M).repeat(N, 1, 1).bool()
x[mask] = value
</code></pre>

Usman_Mahmood: <p>Thank you for the help.</p>

Zhixin_Ma: <p>A simpler way using less memory:</p>
<pre><code class="lang-auto">N, M, value = 5, 3, 0
x = torch.rand(N, M, M)
i = torch.arange(M)
x[:, i, i] = value
</code></pre>
Hybrid CNN + ViT approach

pretbc: <p>Hello, I would like to create hybrid arch CNN + ViT image classification model.</p>
<p>What I wanna do:</p>
<ol>
<li>Extract features from CNN i.e: pretrained EfficientNet_B3</li>
<li>Pass features to (custom) ViT as input of patch embedding</li>
<li>Perform classification</li>
</ol>
<p> → And my question is:<br>
<strong>Whether my approach is appropriate:</strong></p>
<p>code:<br>
Ad1. take last features Conv2NormActivision from EfficenitNet</p>
<pre><code class="lang-auto">│    └─Conv2dNormActivation (8)                   [32, 384, 7, 7]      [32, 1536, 7, 7]              
│    │    └─Conv2d (0)                                     [32, 384, 7, 7]      [32, 1536, 7, 7]        
│    │    └─BatchNorm2d (1)                           [32, 1536, 7, 7]     [32, 1536, 7, 7]       
│    │    └─SiLU (2)                                         [32, 1536, 7, 7]     [32, 1536, 7, 7]
</code></pre>
<p>so I will have <code> [32, 1536, 7, 7]</code></p>
<p>then<br>
Ad2. pass to custom ViT</p>
<pre><code class="lang-auto">patch_embedding_layer = PatchEmbedding(in_channels=1536,
                                        patch_size=1,
                                        embedding_dim=768)

class PatchEmbedding(nn.Module):
    def __init__(self,
                 in_channels: int = 3,
                 patch_size: int = 16,
                 embedding_dim: int = 768):
        super().__init__()
        self.patcher = nn.Conv2d(in_channels=in_channels,
                                 out_channels=embedding_dim,
                                 kernel_size=patch_size,
                                 stride=patch_size,
                                 padding=0)
        self.flatten = nn.Flatten(start_dim=2,  end_dim=3)
    def forward(self, x):
        image_resolution = x.shape[-1]
        assert image_resolution % patch_size == 0, f"Input image size must be divisble by patch size, image shape: {image_resolution}, patch size: {patch_size}"
        x_patched = self.patcher(x)
        x_flattened = self.flatten(x_patched)
        return x_flattened.permute(0, 2, 1) 

</code></pre>
<p>Create new Model based on above:</p>
<pre><code class="lang-auto">class EffC(nn.Module):
    def __init__(self) -&gt; None:
        super(EffC, self).__init__()

        weights = torchvision.models.EfficientNet_B3_Weights.DEFAULT
        model = torchvision.models.efficientnet_b3(weights=weights).to(device)
        for param in model.features.parameters():
            param.requires_grad = False
        self.modelEF = model
        self.vit = ViT(num_classes=len(class_names))

    def forward(self, x):
        x = self.modelEF.features(x)
        x = self.vit(x)
        return x
</code></pre>
Is it possible to change the dimensions of torch::jit::script::Module using only Libtorch (C++)?

ch2cl2: <p>I want to perform transfer learning using Libtorch (C++).<br>
At that time, I think about using a trained model (for example, 1000 classifications of Resnet50), but I want to change the number of output classifications each time.</p>
<p>Please tell me how to change the dimension of the final layer of torchscript only with Libtorch for testing in a C++ environment without Python.</p>
<p>As long as you can save the dimension-changed model with jit::model, I can handle the rest.</p>
<p>that’s all. thank you.</p>
Adam optimizer with jit::script::Module

Nick95: <p>Dear all,<br>
I have successfully imported a pre-trained model in libtorch using the torch::jit::load function:</p>
<pre><code class="lang-auto">torch::jit::script::Module model = torch::jit::load('mymodel.pt');
</code></pre>
<p>Now I want to perform fine-tuning at runtime using the Adam optimizer. I have tried to instantiate the optimizer with:</p>
<pre><code class="lang-auto">torch::optim::Adam optimizer(model.parameters(), torch::optim::AdamOptions(/*lr=*/0.001));
</code></pre>
<p>However, I get error since parameters() returns an object of type <code>torch::jit::parameter_list</code> instead of <code>std::vector&lt;Tensor&gt; params</code>. How can I perform such conversion or solve the problem?</p>
<p>Thanks in advance</p>

tdomhan: <p>I have the same problem. Did you find a solution by any chance?</p>

tdomhan: <p>Something like this seems to work:</p>
<pre><code class="lang-auto">    auto model = torch::jit::load(model_path);
    torch::jit::parameter_list params = model.parameters();
    std::vector&lt;at::Tensor&gt; paramsVector;
    std::transform(params.begin(), params.end(), std::back_inserter(paramsVector),
                   [](const torch::jit::IValue&amp; ivalue) { return ivalue.toTensor(); });
    torch::optim::Adam optimizer(paramsVector, torch::optim::AdamOptions(2e-4)); 
</code></pre>
Routing around link failures in a cluster when training with NCCL backend

xsyourpal: <p>When running FSDP or DDP training on a multi-node NVIDIA DGX cluster with 8 GPUs per node,  standard DGX network configuration which is ,  GPU-GPU has 8 network interfaces (backend) and each node has 2 additional network interfaces (frontend)</p>
<p>is it possible for NCCL or PyTorch Distributed mechanism to route around a link failure on one of the backend network interfaces ?  Does such a failure require restarting the job from a saved checkpoint ?</p>
<p>I am unable to find relevant documentation on fault tolerance related to Network Link Failure recovery or re-routing of traffic. Greatly appreciate pointers.</p>
pytorch implementation of an encoder-decoder with attention for neural machine translation

lord_vader: <p>Hello can you tell me how can I implement this model in pytorch ? I already implemented some models like this before but without attention and now I want to implement this in pytorch . I have several questions and problems but mostly with how can I handle the data flow to both Encoder and the pre-attention decoder .<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/6/9613b208654c13055032a06804bcf8e8b2c60dfa.png" data-download-href="https://discuss.pytorch.org/uploads/default/9613b208654c13055032a06804bcf8e8b2c60dfa" title="NMTModel"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/6/9613b208654c13055032a06804bcf8e8b2c60dfa_2_262x500.png" alt="NMTModel" data-base62-sha1="lpDOHPzbgYIZgEmYHnslBObZr18" width="262" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/6/9613b208654c13055032a06804bcf8e8b2c60dfa_2_262x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/6/9613b208654c13055032a06804bcf8e8b2c60dfa_2_393x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/6/9613b208654c13055032a06804bcf8e8b2c60dfa_2_524x1000.png 2x" data-dominant-color="F6F5F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">NMTModel</span><span class="informations">751×1431 73.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
How to implement LSTM layer with multiple cells per memory block in Pytorch?

Wmog: <p>I intend to implement an LSTM in Pytorch with multiple memory cell blocks - or multiple LSTM units, an LSTM unit being the set of a memory block and its gates - per layer, but it seems that the base class <code>torch.nn.LSTM</code> enables only to implement a multi-layer LSTM with one LSTM unit per layer:</p>
<pre><code>lstm = torch.nn.LSTM(input_size, hidden_size, num_layers)
</code></pre>
<p>where (from the Pytorch’s documentation):</p>
<ul>
<li>
<code>input_size</code> is the input dimension of the network,</li>
<li>
<code>hidden_size</code> is the hidden state dimension for every layer (i.e. the dimension of every layer),</li>
<li>
<code>num_layer</code> is the number of layers of the network</li>
</ul>
<p>Thereupon, from above, each LSTM unit has exactly one cell (the cell state for each LSTM unit is thus a scalar) because for each layer the dimension of the cell state corresponds to the dimension of the hidden state (i.e. <code>hidden_size</code>).</p>
<p>However in the original LSTM model proposed by Hochreiter and Schmidhuber<br>
[1997], each LSTM block/unit can contains several cells:<br>
<a href="https://qph.ec.quoracdn.net/main-qimg-3714ad776948339181a2484495a6e428" rel="nofollow noopener">LSTM Network [Hochreiter, 1997]</a></p>
<p>Is there a way to do so?</p>

codingGoose: <p>I am also dealing with this issue of creating a multi-cell LSTM in PyTorch. Have you been able to solve it?</p>
Is it possible to create a ScriptModule for torchvision Retinanet model?

bradsteiner: <p>Hi, I am using <code>retinanet_resnet50_fpn_v2</code> as defined here: <a href="https://github.com/pytorch/vision/blob/main/torchvision/models/detection/retinanet.py" rel="noopener nofollow ugc">https://github.com/pytorch/vision/blob/main/torchvision/models/detection/retinanet.py</a></p>
<p>I want to convert it to ONNX and use dynamic inputs and outputs, and I saw that using Scripting is recommended for this.<br>
<a href="https://pytorch.org/docs/stable/onnx.html#tracing-vs-scripting" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/onnx.html#tracing-vs-scripting</a></p>
<p>However when I try this like <code>script_module = torch.jit.script(model)</code>, I get the following error:</p>
<pre><code class="lang-auto">...
  File "/lib/python3.9/site-packages/torch/onnx/utils.py", line 1529, in _export
    graph, params_dict, torch_out = _model_to_graph(
  File "/lib/python3.9/site-packages/torch/onnx/utils.py", line 1111, in _model_to_graph
    graph, params, torch_out, module = _create_jit_graph(model, args)
  File "/lib/python3.9/site-packages/torch/onnx/utils.py", line 967, in _create_jit_graph
    module, params = _C._jit_onnx_list_model_parameters(freezed_module)
RuntimeError: 
Unknown type __torch__.torchvision.models.detection._utils.BoxCoder (of Python compilation unit at: 0x5bb5610) encountered in handling model params. This class type does not extend __getstate__ method.:
</code></pre>
<p>Should Scripting be possible for this model?</p>
Loss involving both gradients wrt inputs and network parameters

Hanto322: <p>I have a neural network $F^{\theta}(x)$ characterized by parameters $\theta$, and I am trying to update the network parameters using the following loss function<br>
$$<br>
\frac{ \nabla_{\theta} F^{\theta}(x) }{ \nabla_{x} F^{\theta}(x) } \bigg|_{x = inputs},<br>
$$<br>
i.e. the gradient of $F$ wrt network parameters divided by the gradient of $F$ wrt its inputs. I have started implementing different approaches, which I believe lead to the same results:</p>
<pre><code class="lang-auto">model.optimizer.zero_grad()
x = inputs.clone().requires_grad_()
y1 = model(x)
# ... do one of the 3 approaches ...
loss.backward()
model.optimizer.step()

##### approach 1
grad_y_inputs = torch.autograd.grad(inputs=x,
                                    outputs=y1.sum(),
                                    grad_outputs=None)[0].detach()
y2 = model(inputs)
loss = y2 / grad_y_inputs
#####

##### approach 2
y1.sum().backward(create_graph=True)
loss = y1 / x.grad
#####

##### approach 3
y1.sum().backward()
y2 = model(inputs)
loss = y2 / x.grad
#####
</code></pre>
<ol>
<li>First of all, are the three approaches depicted here doing the desired gradient computation?</li>
<li>Is there any preferred approach, e.g. computational or semantic issues?</li>
</ol>
Does Kaiming He initialisation preserve variance?

niz: <p>As far as I know He initialisation was made to preserve the variance between the inputs and outputs. I noticed that the <code>init.kaiming_normal_()</code> weight initialisation of a <code>Linear</code> layer was not preserving the variance. I gave the layer a normal dataset with a variance of 1.0, passed it though a rectifier, and it outputted a dataset with a variance of roughly 0.7-0.8. I got similar results using <code>init.kaiming_uniform_()</code>. Am I missing something?</p>
<p>I ask this because I have been experiencing an exploding gradient when I add multiple layers to my network.</p>
<p>Here is my code:</p>
<pre><code class="lang-auto">#A layer class with He initialisation
class LinearHe(nn.Linear):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        torch.nn.init.kaiming_normal_(self.weight, a=0.2)

#A Linear layer with 1024 input and output features
linear1 = LinearHe(1024, 1024) 
leaky_relu = nn.LeakyReLU(negative_slope=0.2)

#A normally distributed dataset of size 1024 with mean=0, std=1
a = torch.normal(torch.zeros(1, 1024), torch.ones(1, 1024))
a
&gt;&gt;&gt; tensor([[-1.8505,  0.7651,  1.8227,  ..., -0.3863,  0.6085,  0.6416]])
torch.var(a)
&gt;&gt;&gt; tensor(0.9165)

b = leaky_relu(linear1(a))
torch.var(b)
&gt;&gt;&gt; tensor(0.7133, grad_fn=&lt;VarBackward0&gt;)
#This is a repeating result, the variance is scaled by a factor of roughly 0.8
torch.mean(b)
&gt;&gt;&gt; tensor(0.4346, grad_fn=&lt;MeanBackward0&gt;)
</code></pre>
Matrix rank using vmap

BartekK: <p>Hello,<br>
I am wondering if it is possible to compute the rank of a matrix of a list of tensors with different shapes using vmap.<br>
Every attempt I make ends in failure.</p>
<p>Please help.</p>

BartekK: <p>I found promising avenue using torch.nested.nested_tensor, but the following pops up:</p>
<p>RuntimeError: Internal error: NestedTensorImpl doesn’t support sizes. Please file an issue on <a href="https://github.com/pytorch/nestedtensor" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pytorch/nestedtensor: [Prototype] Tools for the concurrent manipulation of variably sized Tensors.</a></p>
<p>Unfortunately, the given repository is already archived.</p>
How to append a large tensor in order to avoid cpu/gpu crashed

J_B_28: <p>Hello, I am trying to append very large tensor whose dimension is (2000000, 128, 768) during for loop then store it to the disk, I tried to use append, torch.cat and also created an empty tensor first then did slicing, they all cause my code crashed. I am wondering if there are any other ways to achieve that.</p>
<p>This is my current code:<br>
(1) use append:</p>
<pre><code class="lang-auto">feat = []
 with torch.no_grad():

        for test_input, test_label in test_dataloader:
            test_label = test_label.to(device)
            mask = test_input['attention_mask'].to(device)
            input_id = test_input['input_ids'].squeeze(1).to(device)

            output, feat_test = model(input_id, mask)

            feat.append(feat_test.cpu())
            
            del feat_test

torch.save(torch.cat(feat, dim=0), f'feature.pt')
</code></pre>
<p>(2) use torch.cat:</p>
<pre><code class="lang-auto">feat = None
 with torch.no_grad():

        for test_input, test_label in test_dataloader:
            test_label = test_label.to(device)
            mask = test_input['attention_mask'].to(device)
            input_id = test_input['input_ids'].squeeze(1).to(device)

            output, feat_test = model(input_id, mask)

            if feat is None:
               feat = feat_test.cpu()
            else:
               feat = torch.cat((feat, feat_test.cpu()), dim=0, out=feat)
            
            del feat_test

torch.save(feat, f'feature.pt')
</code></pre>
<p>(3) use slicing:</p>
<pre><code class="lang-auto">feat = torch.zeros(2000000, 128, 768)
batch_num = 0
 with torch.no_grad():

        for test_input, test_label in test_dataloader:
            test_label = test_label.to(device)
            mask = test_input['attention_mask'].to(device)
            input_id = test_input['input_ids'].squeeze(1).to(device)

            output, feat_test = model(input_id, mask)
            
            feat[batch_num : batch_num + test_batch_size,:,:] = feat_test.cpu()
            
            del feat_test
            
            batch_num = batch_num + test_batch_size

torch.save(feat, f'feature.pt')
</code></pre>
<p>Any help would be appreciated.</p>

J_B_28: <p>Still have the problem, are there any ways to solve this issue? Thanks.</p>
What does the asterisk * in the PyTorch docs mean?

niz: <p>I have been reading the PyTorch documentation and have come across the asterisk being used as a function argument. I know that the asterisk is used in Python to pack/unpack arguments, but I can not find any reference to this case online.</p>
<p>An example of this is the <code>torch.normal</code> function. The <a href="https://pytorch.org/docs/stable/generated/torch.normal.html" rel="noopener nofollow ugc">documentation</a> gives the following with no explanation of the asterisk:</p>
<blockquote>
<p>torch.normal(<em>mean</em>, <em>std</em>, *, <em>generator=None</em>, <em>out=None</em> )</p>
</blockquote>

arunppsg: <p>If I am not wrong, the arguments after <code>*</code> are keyword only arguments, introduced in <a href="https://peps.python.org/pep-3102/" rel="noopener nofollow ugc">PEP 3102</a>.</p>
Creating a .pt model for sample production that can be used by a C++ program

Patrick_Asenov: <p>Dear all,</p>
<p>I am trying to obtain a trained model flow_model.pt (produced from a modified version of <a href="https://github.com/bayesiains/nflows/blob/master/examples/conditional_moons.ipynb" rel="noopener nofollow ugc">https://github.com/bayesiains/nflows/blob/master/examples/conditional_moons.ipynb</a> ) which can be later read in C++ using a command like:</p>
<p>module = torch::jit::load(“flow_model.pt”);</p>
<p>or something similar.</p>
<p>The important snippets of my initial PyTorch program so far are:</p>
<p>class model_from_flow(torch.nn.Module):<br>
def <strong>init</strong>(self, flow):<br>
super().<strong>init</strong>()<br>
self.flow = flow</p>
<pre><code>def forward(self, x):
    self.flow.eval()
    output_dist = self.flow.sample(num_samples=1, context=x)
    return output_dist
</code></pre>
<h3><a name="h-1" class="anchor" href="#h-1"></a></h3>
<p>x, y = datasets.make_moons(128, noise=.1)<br>
plt.scatter(x[:, 0], x[:, 1], c=y);</p>
<p>num_layers = 5<br>
base_dist = ConditionalDiagonalNormal(shape=[2],<br>
context_encoder=nn.Linear(1, 4))</p>
<p>transforms = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
for _ in range(num_layers):<br>
transforms.append(ReversePermutation(features=2))<br>
transforms.append(MaskedAffineAutoregressiveTransform(features=2,<br>
hidden_features=4,<br>
context_features=1))<br>
transform = CompositeTransform(transforms)</p>
<p>flow = Flow(transform, base_dist)<br>
flow.eval()<br>
optimizer = optim.Adam(flow.parameters())</p>
<p>num_iter = 1000<br>
for i in range(num_iter):<br>
x, y = datasets.make_moons(128, noise=.1)<br>
x = torch.tensor(x, dtype=torch.float32)<br>
y = torch.tensor(y, dtype=torch.float32).reshape(-1, 1)<br>
optimizer.zero_grad()<br>
loss = -flow.log_prob(inputs=x, context=y).mean()<br>
loss.backward()<br>
optimizer.step()</p>
<pre><code>if (i + 1) % 500 == 0:
    fig, ax = plt.subplots(1, 2)
    xline = torch.linspace(-1.5, 2.5, 100)
    yline = torch.linspace(-.75, 1.25, 100)
    xgrid, ygrid = torch.meshgrid(xline, yline, indexing='ij')
    #xyinput = torch.cat([xgrid.reshape(-1, 1), ygrid.reshape(-1, 1)], dim=1)

    with torch.no_grad():
        zgrid0 = flow.sample(5000, context=torch.zeros(1, 1)).exp().reshape(100, 100)
        zgrid1 = flow.sample(5000, context=torch.ones(1, 1)).exp().reshape(100, 100)
</code></pre>
<h1><a name="save-the-model-2" class="anchor" href="#save-the-model-2"></a>Save the model</h1>
<p>wrapped_model = model_from_flow(flow)<br>
wrapped_model.eval()<br>
with torch.no_grad():<br>
fake_input = torch.zeros(128, 1)<br>
traced_model = torch.jit.script(wrapped_model)<br>
traced_model.eval()<br>
traced_model.save(“flow_model.pt”)</p>
<p>I get an error like this:</p>
<pre><code>    raise RuntimeError("Forward method cannot be called for a Distribution object.")
</code></pre>
<p>I have tried using torch.jit.trace too. Can anyone point me in the right direction? What is the proper syntax to save the model?</p>
<p>Thank you in advance.</p>
<p>Best</p>
How to load pretrained weights of swim transformers

Kapil_Rana: <p>from torchvision.models import swin_v2_b,swin_t, swin_b<br>
model = swin_b(weights=“Swin_B_Weights”)</p>
<p>It gives error<br>
KeyError: ‘Swin_B_Weights’</p>

ptrblck: <p>This should work:</p>
<pre><code class="lang-python">from torchvision.models import swin_v2_b,swin_t, swin_b, Swin_B_Weights
model = swin_b(weights=Swin_B_Weights.IMAGENET1K_V1)
</code></pre>
I run the code in ssh server and the server has Xserver but it give me the attachment error: Do anyone know how can I solve this error?

Aya_Elfatyany: <p>the steps I did to run the code using these commands:</p>
<ol>
<li>export QT_QPA_PLATFROM=offscreen</li>
<li>export QT_DEBUG_PLUGINS=1</li>
</ol>
<p>ERR| vtkXOpenGLRenderWindow (0x23e4770): bad X server connection. DISPLAY=<br>
Aborted (core dumped)</p>

ptrblck: <p>As your <a href="https://discuss.pytorch.org/t/pytq5-installing/186985/2">previous post</a> this one is also unrelated to PyTorch, so please post it in a Qt-specific discussion board.</p>