akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
IndexError: tuple index out of range

lavender99: <pre><code class="lang-auto">for num_epochs in range(1):  # loop over the dataset multiple times

    running_loss = 0.0
    for i, data in enumerate(train_loader, 0):
        # get the inputs
        inputs, labels = data

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = hh(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item()
        if i % 2000 == 1999:    # print every 2000 mini-batches
            print('[%d, %5d] loss: %.3f' %
                  (epoch + 1, i + 1, running_loss / 2000))
            running_loss = 0.0

print('Finished Training')
</code></pre>
<p>in the code above when i run it gives me " IndexError: tuple index out of range "<br>
i can see that the tensor size is zero here but  i do nor know why</p>

ptrblck: <p>Which line is throwing this error?<br>
I’m not sure where you are indexing.</p>

gaalocastillo: <p>Maybe it is similar to me. I am getting the same error in the for loop over the training dataloader. In my code it is happening exactly in the for line:</p>
<pre><code class="lang-auto">def train_model(model, optim, train_dl, class_weights=None):
  ...
  for x1, x2, y, source_apps, event_app_lists, target_app_lists, sw in train_dl:
    ...
</code></pre>
<p><code>event_app_lists</code> and <code>target_app_lists</code> are packed sequences.</p>

ptrblck: <p>Could you post a minimal and executable code snippet I could use to reproduce and debug the issue, please?</p>
Why the output of torch.rand() keeps only four decimal places

kyjsj: <p>Hi all,<br>
I’m currently working on a project related to diffusion models and I have encountered an issue with the <code>torch.rand()</code> function. I have noticed that the output of this function only has four decimal places, even when multiplied by certain numbers. I have checked the documentation for this function but I am still confused about why this is happening.<br>
Can anyone offer any insights into this issue? Thanks!<br>
Some examples:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; a = torch.rand(1)
&gt;&gt;&gt; print(a)
tensor([0.6181])
&gt;&gt;&gt; a * 0.999
tensor([0.6175])
&gt;&gt;&gt; a * 0.001
tensor([0.0006])
&gt;&gt;&gt; a * 0.0001
tensor([6.1810e-05])
&gt;&gt;&gt; a * 0.9991
tensor([0.6175])
&gt;&gt;&gt; a * 0.99991
tensor([0.6180])
</code></pre>

jiversivers: <p>This is just how the tensors are displayed. If you <code>print(a.item())</code>, you should see more figures. You can force the data type in <code>torch.rand()</code>, too, with the keyword argument <code>dtype = torch.&lt;your desired datatype, eg. float64&gt;</code> which can allow for even more significant figures, if you’d like.</p>
<p>Check out <a href="https://discuss.pytorch.org/t/how-to-print-more-than-4-digits-after-decimal/63405">this topic</a> if you want to always print your tensors with more figures after the decimal shown.</p>
How to create an artificial index for MaxUnpool2D

Qikai_Lu: <p>I am currently working on an asymmetric autoencoder (encoder and decoder have different architectural designs), and would like to use MaxUnpool2D for the decoder. However, the module requires an ‘indices’ argument, which I cannot obtain from the encoder-side. I am wondering how do I efficiently create an artificial ‘indices’? Furthermore, if the ‘indices’ creation process can be done batchwise, that would be even better.</p>
<p>Thanks in advance.</p>

ptrblck: <p>Take a look at this example:</p>
<pre><code class="lang-python">pool = nn.MaxPool2d(2, return_indices=True)

x = torch.arange(2*16).view(2, 1, 4, 4).float()
print(x)
# tensor([[[[ 0.,  1.,  2.,  3.],
#           [ 4.,  5.,  6.,  7.],
#           [ 8.,  9., 10., 11.],
#           [12., 13., 14., 15.]]],


#         [[[16., 17., 18., 19.],
#           [20., 21., 22., 23.],
#           [24., 25., 26., 27.],
#           [28., 29., 30., 31.]]]])

out, idx = pool(x)
print(out)
# tensor([[[[ 5.,  7.],
#           [13., 15.]]],


#         [[[21., 23.],
#           [29., 31.]]]])

print(idx)
# tensor([[[[ 5,  7],
#           [13, 15]]],


#         [[[ 5,  7],
#           [13, 15]]]])

unpool = nn.MaxUnpool2d(2)
y = unpool(out, idx)
print(y)
# tensor([[[[ 0.,  0.,  0.,  0.],
#           [ 0.,  5.,  0.,  7.],
#           [ 0.,  0.,  0.,  0.],
#           [ 0., 13.,  0., 15.]]],


#         [[[ 0.,  0.,  0.,  0.],
#           [ 0., 21.,  0., 23.],
#           [ 0.,  0.,  0.,  0.],
#           [ 0., 29.,  0., 31.]]]])
</code></pre>
<p>to understand what the returned indices from an <code>nn.MaxPool2d</code> layer represent and how you could create “artificial indices”.<br>
I also don’t fully understand what your use case would be if you want to randomly create these indices.</p>

Qikai_Lu: <p>Thanks. That helps.</p>
<p>For the use case, I am working with an asymmetric autoencoder. Specifically, the encoder is a pretrained black-box (of some unknown architecture) that generates a latent vector for each input image. I need to design an decoder that can convert the latent vector into a reconstructed image. Based on previous knowledge, MaxUnpool2D and ConvTranspose2D can be used in the decoder design.</p>
<p>However, if there are better solutions (papers, tutorial, etc.) that avoids constructing an artificial idx, please let me know.</p>
Quadruple precision complex float

steven_morad: <p>I am dealing with very large and small numbers that can sometimes overflow/underflow a <code>torch.dtype.complex128</code>. Specifically, I am computing something resembling a discrete Laplace transform over a large window</p>
<pre><code class="lang-python">t = torch.arange(10_000).reshape(1, -1)
s = torch.rand(torch.zeros(5, 1), 1.0)
torch.cumsum(f(x) * torch.exp(-t * s), dim=-1)
</code></pre>
<p>I’m aware that torch does not currently support quadruple precision floats. What would be the easiest way to do this operation with autograd support? Would I need to write it up in CUDA – does CUDA even support complex quads? Should I do some trickery and explicitly represent the mantissa and exponent as torch floats, rewriting basic operations like addition/multiplication?</p>
<p>What options do I have?</p>
How To Solve EasyCLA Issue

chi_hiwhy: <p>I contribute new loss function into Pytorch but show that<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/f/ff1b55e7f3208affda076fc61ae5195d8a13dd8a.png" data-download-href="https://discuss.pytorch.org/uploads/default/ff1b55e7f3208affda076fc61ae5195d8a13dd8a" title="Screenshot from 2023-07-12 18-13-00"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ff1b55e7f3208affda076fc61ae5195d8a13dd8a_2_690x224.png" alt="Screenshot from 2023-07-12 18-13-00" data-base62-sha1="AoMjUw33PGSKnm2VTEtFPj6FAgy" width="690" height="224" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ff1b55e7f3208affda076fc61ae5195d8a13dd8a_2_690x224.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ff1b55e7f3208affda076fc61ae5195d8a13dd8a_2_1035x336.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ff1b55e7f3208affda076fc61ae5195d8a13dd8a_2_1380x448.png 2x" data-dominant-color="F4F0F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2023-07-12 18-13-00</span><span class="informations">1916×623 86.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
error :</p>
<p>please can help me how to solve this issue?</p>

ptrblck: <p>Did you check the linked articles and tried to resolve it?</p>

chi_hiwhy: <p>I already check but not work. And also my GitHub accounts have two email addresses added.</p>

ptrblck: <p>In case you added the other email address recently to your GitHub account you might need to rerun the CLA check. If that doesn’t help try to tag <code>zpao</code> in your PR as he could try to help assuming the suggested solutions shown in the links don’t work.</p>

chi_hiwhy: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="184037">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>to rerun the CLA check.</p>
</blockquote>
</aside>
<p>Yes, it’s work, and thank you so much for your help.</p>
Transfer learning low accuracy and CUDA not working

beazt: <p>Hi there,</p>
<p>I am running into a few problems while trying out this code:</p><aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/examples/blob/main/cpp/transfer-learning/main.cpp">
  <header class="source">

      <a href="https://github.com/pytorch/examples/blob/main/cpp/transfer-learning/main.cpp" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/examples/blob/main/cpp/transfer-learning/main.cpp" target="_blank" rel="noopener nofollow ugc">pytorch/examples/blob/main/cpp/transfer-learning/main.cpp</a></h4>


      <pre><code class="lang-cpp">//
//  main.cpp
//  transfer-learning
//
//  Created by Kushashwa Ravi Shrimali on 12/08/19.
//

#include "main.h"

torch::Tensor read_data(std::string location) {
    /*
     Function to return image read at location given as type torch::Tensor
     Resizes image to (224, 224, 3)
     Parameters
     ===========
     1. location (std::string type) - required to load image from the location
     
     Returns
     ===========
     torch::Tensor type - image read as tensor
</code></pre>



  This file has been truncated. <a href="https://github.com/pytorch/examples/blob/main/cpp/transfer-learning/main.cpp" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>So basically I have just downloaded the git and the dog and cat samples and run the code without any changes, except for the paths I am using of course. The first problem is, that I only get accuracies of about 51-52% which does not go higher after 25 epochs. Do I need to set up something different to get higher accuracies?</p>
<p>While classifying, I only get about 15-30% accuracies which is not acceptable I guess.</p>
<p>Another problem I have is, when I try to train on CUDA, which runs just fine and I can classifying using CUDA but the accuracies are negative like so: -35940% for example.</p>
<p>The changes I did for training was this:</p>
<pre><code class="lang-auto"> module = torch::jit::load(weight, at::kCUDA);
 lin-&gt;to(at::kCUDA);
 target = target.to(at::kCUDA);
 input.push_back(data.to(at::kCUDA));
</code></pre>
<p>and for the classifying part I made these changes:</p>
<pre><code class="lang-auto"> img_tensor = img_tensor.to(at::kCUDA);
 model = torch::jit::load(model_path, at::kCUDA);
 model.eval();
 model_linear-&gt;to(at::kCUDA);
 torch::load(model_linear, model_path_linear, at::kCUDA);
</code></pre>
<p>Anybody can help please?</p>
<p>Edit: The classifying does not work at all. While using the CPU trained version, I always get about the same results, which is about 20% cat, 30% dog, it does only change a few % whatever images I am using to classify. The results always stay about the same.</p>
Error Building LibTorch

alexcorcia: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/8/58f95891fdb5b538a4b970df9a24fadc646bfb21.png" data-download-href="https://discuss.pytorch.org/uploads/default/58f95891fdb5b538a4b970df9a24fadc646bfb21" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/8/58f95891fdb5b538a4b970df9a24fadc646bfb21.png" alt="image" data-base62-sha1="cH6eNxblubPi18uV69ROUlO59zH" width="690" height="113" data-dominant-color="3C3D3F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1078×178 9.28 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I haven’t touched any of the code but I have this issue. Any idea ? I’m just trying to add PyTorch to my personal project</p>
<p>I have taken everything from the most simple example Pytorch has and I made it work outside my project on its own but without touching the code it now breaks …</p>
Cannot import name 'tensormask' from 'detectron2.projects'

RadAlienware: <p>I followed the setup of <a href="https://github.com/facebookresearch/detectron2/tree/main/projects/TensorMask" rel="noopener nofollow ugc">https://github.com/facebookresearch/detectron2/tree/main/projects/TensorMask</a>. But still I can not import it. As I can with from detectron2.projects import point_rend.</p>
<p>I run the below line.<br>
<code>pip install -e /path/to/detectron2/projects/TensorMask</code></p>
<p>Still no luck</p>
<pre><code class="lang-auto">ImportError                               Traceback (most recent call last)
Cell In[62], line 1
----&gt; 1 from detectron2.projects import tensormask

ImportError: cannot import name 'tensormask' from 'detectron2.projects' (/kaggle/working/detectron2/detectron2/projects/__init__.py)
</code></pre>
<p>I read some of the solutions, but they are vague. <em>Some suggested that to change the name of detectron2 directory will solve it. But for me it is not.</em></p>
Accumulating Gradients Manually

Qikai_Lu: <p>I am currently trying to implement a Federated Learning algorithm, and have a question about gradient accumulation for model parameters.</p>
<p>To simulate that the server model must receive gradients from all client model (akin to FedSGD) before each update, I have implemented following code:</p>
<pre><code class="lang-auto">optim = optimizer.SGD(network.parameters(), lr=0.001)
optim.zero_grad()

for client in client_list:
    # compute g_client as an output of torch.autograd.grad
    # ...
    for p, g in zip(network.parameters(), g_client):
        if p.grad is None:
            p.grad = g
        else:
            p.grad += g

for p in network.parameters():  # average the gradient
    p.grad /= len(client_list)

optim.step()
</code></pre>
<p>I am wondering if this is a proper way to accumulate the gradients from the different clients before updating my network module? This is my first time doing model backpropagation and weight updates without relying on “loss.backward(); optim.step()”, so I am wondering if there are some under-the-hood mechanisms that I am missing for model training.</p>

Qikai_Lu: <p>Any response or advice would be welcome!</p>
Why does each torch.bool value take up an entire byte?

GuillaumeTong: <pre><code class="lang-auto">In [2]: import torch
In [3]: t = ~torch.empty(100000000, dtype=torch.bool, device="cuda")
In [4]: print(torch.cuda.memory_allocated())
100663296
In [5]: print(100663296/100000000)
1.00663296
</code></pre>
<p>As shown above, it looks like a large <code>torch.bool</code> tensor will take up roughly 1 byte per item (at least on CUDA. I don’t know how to measure this on CPU).</p>
<p>Why is that? I would have assumed that a bool needs only 1 bit to represent, not 8 (making up a byte).</p>

eqy: <p>The size of a bool is implementation dependent, and I think 1 byte is a relatively common choice. See also: <a href="https://github.com/pytorch/pytorch/issues/41571" class="inline-onebox" rel="noopener nofollow ugc">torch.cuda.BoolTensor uses 8 bits per element, not 1 bit as reported by element_size() · Issue #41571 · pytorch/pytorch · GitHub</a></p>

GuillaumeTong: <p>Thanks <a class="mention" href="/u/eqy">@eqy</a>, that link is very helpful.<br>
As a quick summary (and to verify I got this right), looks like this is currently implemented as-is to facilitate in-memory addressing, but some experts are already looking into fixing this, using features developed for newer 2-bit/4-bit dtypes.</p>
<p>In the mean time, as a workaround, given that PyTorch has a set of bitwise logical operator that can work with uint8, we could conceivably re-organize a 8*N sized bool tensor into a N sized uint8 tensor with the following benefits:</p>
<ol>
<li>Using less memory</li>
<li>More efficient computations (maybe?)</li>
</ol>
<p>And the following cons:</p>
<ol>
<li>Indexing/slicing the compressed tensor at the sub-byte level is complicated without de-compression</li>
<li>Some more complex operations like masking will not be supported</li>
</ol>
<p>As for the conversion method, my current best idea would be:</p>
<ul>
<li>converting from “uint8bool” to bool: right shift the values according to indexes, and check for 1 mod 2</li>
<li>converting from bool to “uint8bool”: convert bool to int, left shift according to index, and sum</li>
</ul>
<pre><code class="lang-auto">In [1]: uint8_tensor = torch.randint(0, 255, (1, 2), dtype=torch.uint8)
In [2]: uint8_tensor
tensor([[67, 27]], dtype=torch.uint8)

In [3]: indexes = torch.arange(8).unsqueeze(0).unsqueeze(0)
In [4]: indexes
tensor([[[0, 1, 2, 3, 4, 5, 6, 7]]])

In [5]: bool_tensor = bool_tensor = (uint8_tensor.unsqueeze(-1)&gt;&gt;indexes)%2==1
In [6]: bool_tensor
tensor([[[ True,  True, False, False, False, False,  True, False],
         [ True,  True, False,  True,  True, False, False, False]]])

In [7]: reconverted = (bool_tensor.byte() &lt;&lt; indexes).sum(dim=-1)
In [8]: reconverted
tensor([[67, 27]])

In [9]: uint8_tensor[0,0] | uint8_tensor[0,1]
tensor(91, dtype=torch.uint8)
In [10]: 1+2+8+16+64
91
In [11]: uint8_tensor[0,0] &amp; uint8_tensor[0,1]
tensor(3, dtype=torch.uint8)
</code></pre>
<p>The “uint8bool” has the most significant bit at the end, which might not be what most people prefer. In that case, inverting <code>indexes</code> should be enough.</p>
Error on passing GPT-NEO-1.3B into DDP[Error: Invalid Scalar Type]

iammano: <p>Hi there,</p>
<p>Problem:<br>
Unable to wrap the model inside DDP.</p>
<p>Note:<br>
I’m using gloo backend with DDP. (Does not have GPU)</p>
<p>Spec:</p>
<ol>
<li>24 cores CPU</li>
<li>32GB RAM</li>
<li>512 GB SSD</li>
</ol>
<p>I already deployed model training with DDP(multinode) using the basic CNN class.</p>
<p>I’m facing the error <strong>“RuntimeError: Invalid scalar type”</strong> upon supplying the model(GPT-NEO-1.3B) into the DDP wrapper</p>
<p>class Agent():</p>
<pre><code>def __init__(self, model, dataset: DataLoader):
    self.local_rank = int(os.environ["LOCAL_RANK"])
    self.global_rank = int(os.environ["RANK"])
    self.dataset = dataset
    self.model = DDP(model) #This is the line I'm getting the error

def train(self):
    #Re-Train the model
    training_args = TrainingArguments(output_dir="./output", num_train_epochs=1, logging_steps=10, 
                             save_strategy="epoch", per_device_train_batch_size=2, per_device_eval_batch_size=2,
                             warmup_steps=100, weight_decay=0.01, logging_dir="./logs")
    Trainer(model=self.model, args=training_args, train_dataset = self.dataset, data_collator=lambda data:{
            "input_ids":torch.stack([f[0] for f in data]).to(f"cpu:{local_rank}"),
            "attention_mask":torch.stack([f[1] for f in data]).to(f"cpu:{local_rank}"),
            "labels":torch.stack([f[0] for f in data]).to(f"cpu:{local_rank}")
        }).train()
</code></pre>
<p>This is my class, apart from that I have initialized the DDP process group and wrapped the dataset into a DistrbutedSampler Inside the Dataloader sampler parameter.</p>
<p>Kindly help me out, thanks in advance.</p>

Alex_Smith1: <p>I’m facing a similar issue.</p>

iammano: <p>Hi <a class="mention" href="/u/alex_smith1">@Alex_Smith1</a> i resolved the issue, the thing here is DDP does not accept torch.bool type tensors. Here gpt neo’s buffer in torch.bool type i converted the buffers of the model into torch.uint8 . Try this might be helpful.</p>

Alex_Smith1: <p>Thanks <a class="mention" href="/u/iammano">@iammano</a>. That seems to have resolved the error but looks like uint8 is deprecated for attention tensors and I get the following warning. It’s a little surprising that bool is not supported</p>
<p>/site-packages/torch/nn/functional.py:5080: UserWarning: Byte tensor for attn_mask in nn.MultiheadAttention is deprecated. Use bool tensor instead.<br>
warnings.warn(“Byte tensor for attn_mask in nn.MultiheadAttention is deprecated. Use bool tensor instead.”)</p>

iammano: <p>Yes <a class="mention" href="/u/alex_smith1">@Alex_Smith1</a> if you try with int8 or something we get error like expected torch.bool type but got char dtype, i tried that. This error will triggered when we provide data to the model.</p>
<p>But uint8 works.</p>
Cosine embedding loss

Anesh_Muthiah_A: <p>I am using cosine embedding loss. This is the model</p>
<pre><code class="lang-auto">class CNN(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers, num_classes):
        super(CNN, self).__init__()
        
        self.num_layers = num_layers
        self.hidden_size = hidden_size
        #self.hidden_size2-hidden_size2
        
        # 1D Convolutional layers
        self.cnn1 = nn.Conv1d(input_size, hidden_size, kernel_size=5, stride=1, padding=1)

        self.dropout=nn.Dropout(0.3)
        # Max pooling layer
        self.maxpool = nn.MaxPool1d(kernel_size=2, stride=1)
        # LSTM layer
       
        # Fully connected layer
        self.fc1 = nn.Linear(864, 128)
 
        self.act1=nn.ReLU()
        self.cnn=torch.nn.Sequential(self.cnn1,self.act1,self.maxpool)

        #self.act2=nn.Tanh()
#         self.act3=nn.ReLU()
#         self.act4=nn.Tanh()
#         self.act5=nn.Tanh()
        
    def forward(self, inp1,inp2):

        out1=self.cnn(inp1.transpose(1, 2))
        out2=self.cnn(inp2.transpose(1, 2))
        
        
        out1 = out1.transpose(1, 2)  # Transpose back to (batch_size, sequence_length, channels)
        out2 = out2.transpose(1, 2)
        out1=torch.reshape(out1,(out1.size(0),-1))
        out2=torch.reshape(out2,(out2.size(0),-1))
        #print(out1)


        # Fully connected layer
        emb1 = self.fc1(out1)  # Use the last timestep's output
        emb2 = self.fc1(out2)

        return emb1,emb2
</code></pre>
<p>I am getting nan from the output of the cnn. It was working first but then it is giving nan. Please help me solve this</p>
DistributedDataParallel: using torch.cuda

rooted: <p>Can anyone please help to resolve this error</p>
<p>I have the model defined as</p>
<p>model = DDP(model, device_ids=[torch.distributed.get_rank() % torch.cuda.device_count()])</p>
<p>ValueError: DistributedDataParallel device_ids and output_device arguments only work with single-device/multiple-device GPU modules or CPU modules, but got device_ids [0], output_device None, and module parameters {device(type=‘cpu’)}</p>

ptrblck: <p>Did you try to move the <code>model.to(rank)</code> as it seems it’s still stored on the CPU?</p>

rooted: <p>Thanks for the response</p>
<p>Yes I added this line instead</p>
<p>model = model.to(‘cuda’)</p>
<p>before wrapping the DDP around it which fixed the error</p>
Tensor assembling utility like numpy.block

abhisek: <p>I am looking for a generic way to construct a block tensor from a specified arrangement of multiple tensors. For example, constructing the tensor <code>[[A, B], [C, D]] </code>where <code>A</code>, <code>B</code>, <code>C</code>, <code>D</code> are all tensors. In <code>numpy</code> simply <code>np.block( [[A, B], [C, D]])</code> does the trick. Is there any similar function in <code>pytorch</code> to achieve the same?</p>
<p>Doc for <code>numpy.block</code>: <a href="https://numpy.org/doc/stable/reference/generated/numpy.block.html" rel="noopener nofollow ugc">numpy.block - manual</a></p>
Plotting the Loss vs Epoch graph for my neural network

Brooklyn_Amaro: <p>I am trying to plot my loss vs epoch graph to determine a good number of epochs to use but I am coming across a graph that looks like this and I don’t know how to fix it.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/a/0a1015a7aa0db85ccaae603f3fc531ab6fcbfc21.png" alt="image" data-base62-sha1="1r1eCkGwW7yHTXPEFcLPkQuiVRn" width="386" height="259"></p>
<p>Here is my code:</p>
<pre><code class="lang-auto">import torch
import numpy as np
from torch import nn
from torch import optim
import random
from torch.utils.data import TensorDataset
from torch.utils.data import DataLoader
import pandas as pd
import matplotlib.pyplot as plt


# Creating Simple Data
def f(x):
    y = (x**2) + 2
    return y

X = []
for i in range(60):
    X.append(random.randint(0,500))

y = []    
for x in X:
    y.append(f(x))
    
X = np.array(X)
y = np.array(y)

split = int(0.8*len(X))
X_train, X_test = X[:split], X[split:]
y_train, y_test = y[:split], y[split:]

X_train, X_test = torch.Tensor(X_train), torch.Tensor(X_test)
y_train, y_test = torch.Tensor(y_train), torch.Tensor(y_test)


train_dataset = TensorDataset(X_train, y_train)
train_dataloader = DataLoader(train_dataset, batch_size = 1)
test_dataset = TensorDataset(X_test, y_test)
test_dataloader = DataLoader(test_dataset, batch_size = 5)
    


device = 'cuda' if torch.cuda.is_available() else 'cpu'

input_dim = 1
hidden_dim = 1
output_dim = 1

class NeuralNetwork(nn.Module):
    def __init__(self, input_dim, hidden_dim, output_dim):
        super(NeuralNetwork, self).__init__()
        
        # Define and instiante layers
        self.hidden1 = nn.Linear(input_dim,hidden_dim)
        self.hidden_activation = nn.ReLU() 
        self.out = nn.Linear(hidden_dim, output_dim)
        
        # Defines what order to inputs will go into each layer
    def forward(self, x):
        y = self.hidden1(x)
        y = self.hidden_activation(x)
        # no activation function for output of a regression problem
        y = self.out(x)
        return y
    
    
# Creating model, putting model in neural network
model = NeuralNetwork(input_dim, hidden_dim, output_dim).to(device)

learning_rate = 1.e-6
loss_fn = nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)
       
epochs = 10
loss_values = []
for epoch in range(epochs):
    ##### Training Data #####
    model.train()
    train_loss = 0
     
    for i, (X, y) in enumerate(train_dataloader):
        X, y = X.to(device), y.to(device)                 
        y_pred = model(X)
        
        loss = loss_fn(y_pred, y)
        train_loss += loss.item() 
        
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        
        loss_values.append(train_loss/ len(train_dataset))
        
plt.plot(loss_values)</code></pre>
Don't use gpu but cpu·

Ljtyxhc: <p>i use cuda 11.6 with pytorch 1.13, the question is very amazing!!!<br>
the OS is ubuntu 18.04.6 with the GPU TITAN2080 Driver Version: 510.47.03<br>
sometimes when i start a program, it will only use cpu instead of gpu, however, i can promise that the code is ok because it worked after i restart it</p>
<p>part of code are following:</p>
<pre><code class="lang-auto">with torch.no_grad():
    for file in data_loader:
        start = time.time()
        label, pic = file # return label and a imagenet pic (24,3,224,224)
        pred = model(pic.cuda()) # with resnet50
        for i in range(pred.shape[0]):
            max_index = pred[i].argmax()
            correct = label[i]
            if max_index == correct:
                ji += 1
        torch.cuda.synchronize()
        end = time.time()
        all_time += end - start

</code></pre>

ptrblck: <aside class="quote no-group" data-username="Ljtyxhc" data-post="1" data-topic="184030">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ljtyxhc/48/62138_2.png" class="avatar"> Ljtyxhc:</div>
<blockquote>
<p>sometimes when i start a program, it will only use cpu instead of gpu</p>
</blockquote>
</aside>
<p>How did you verify that the CPU is only used?</p>
Facing problems with Detectron2 Lazy Config (.py) file

RadAlienware: <p>Hey. So, I want to use a config file from a detectron2 project which is under MViTv2 in my custom dataset which is “my_dataset_train” (coco (datacatalog format)</p>
<p>I want to use this config file<br>
[<a href="https://github.com/facebookresearch/detectron2/blob/main/projects/MViTv2/configs/mask_rcnn_mvitv2_t_3x.py" rel="noopener nofollow ugc">https://github.com/facebookresearch/detectron2/blob/main/projects/MViTv2/configs/mask_rcnn_mvitv2_t_3x.py</a>]</p>
<p>like the beneath typical way I use a yaml config file. But giving errors about trainer when i implement it on lazy config.</p>
<p>I figured some of the config. Here is the structure.</p>
<pre><code class="lang-auto">
import logging
import sys

from detectron2.checkpoint import DetectionCheckpointer
from detectron2.config import LazyConfig, instantiate
from detectron2.engine import (
    AMPTrainer,
    SimpleTrainer,
    DefaultTrainer,
    default_argument_parser,
    default_setup,
    default_writers,
    hooks,
    launch,
)
# from detectron2.engine.defaults import create_ddp_model
from detectron2.evaluation import inference_on_dataset, print_csv_format
from detectron2.utils import comm


cfg = LazyConfig.load("/kaggle/working/detectron2/projects/MViTv2/configs/mask_rcnn_mvitv2_t_3x.py")
cfg.dataloader.train.dataset.names='my_dataset_train'
cfg.optimizer = model_zoo.get_config("common/optim.py")

optim = instantiate(cfg.optimizer)

# train_loader = instantiate(cfg.dataloader.train)
# dataset=instantiate(cfg.dataloader.train)
model = instantiate(cfg.model)
model.to(cfg.train.device)
# cfg.train.amp.enabled=True
cfg.train.max_iter = 100
cfg.optimizer.lr = 0.001
cfg.dataloader.train.total_batch_size = 8
cfg.train.init_checkpoint = "detectron2://ImageNetPretrained/mvitv2/MViTv2_T_in1k.pyth"

# trainer = SimpleTrainer(model, train_loader, optim)
trainer=DefaultTrainer(cfg)

print(model)

trainer.train()
</code></pre>
<p><strong>Error or Issue:</strong>  The issue is with <em><strong>trainer</strong></em>. <em>The AMP trainer is somewhat for parallel distributed system</em>. And it is kinda tough for me to figure it out. But *</p>
<p>default trainer is telling me to add solver but solver is not for lazy config.</p>
<ul>
<li></li>
</ul>
<p>Can you guys help me how to run it in single  gpu? By default it says 64 gpus.</p>
Saving model and Re-Laoding the saved-model

augusmaa: <p>Hello everyone,</p>
<p>I have a question about loading a saved model.<br>
I trained a model on a server with 8 GPU using nn.Dataparallel I got a performance of 78.07% accuracy on the validation set. But I reload the saved model on another server with 2 GPU (still using Dataparallel) to run the validation script (of course with the same dataset), and the performance decreases to 76.86% accuracy. I am wondering why.</p>
<p>Note:  When I go back to the server where the model has been trained everything works normally, I found exactly the same performance by running the validation script.</p>
<p>I would like to know if I did something wrong either in saving the model or in re-loading the saved-model.<br>
If you have any advice that could help will be welcome.</p>
<p>To save the model I used:</p>
<pre><code class="lang-auto">torch.save({
             'epoch':epoch,
             'model_state_dict': model.state_dict(),
             'optimizer_state_dict':optimizer.state_dict(),}, ‘./save_model_folder/model.tar')
</code></pre>
<p>To load the saved model I  used:</p>
<pre><code class="lang-auto">checkpoint = torch.load(‘./save_model_folder/model.tar’)
model.load_state_dict(checkpoint['model_state_dict'])
model.eval()
</code></pre>
<p>Thank you in advance.</p>
Linear multiplication over every element of 2d input?

Joseph_Park: <p>Suppose I have an input like torch.shape = (64, 123)</p>
<p>I want to do a nn.linear(64,1) on each of the rows so I get output shape = (1, 123) Is there a way to do this?</p>

ptrblck: <p>You could transpose the input and output:</p>
<pre><code class="lang-python">x = torch.randn(64, 123)
lin = nn.Linear(64, 1)

out = lin(x.T).T
print(out.shape)
# torch.Size([1, 123])
</code></pre>
<p>but should make sure the dimensions are interpreted correctly.<br>
Note that the linear layer expects inputs in the shape <code>[batch_size, in_features]</code> so check if transposing is creating these dimensions.</p>

Joseph_Park: <p>Dang! It works! Thanks a million.</p>
Pack_padded_sequence output shape is strange

laro: <p>I have the following code:</p>
<pre><code class="lang-auto">gruLayer = nn.GRU(input_size = 4, hidden_size = 32, num_layers = 2, bidirectional=False, batch_first=True)

print(f"{arr_input.shape}, {arr_seq_len.shape}")
pack     = nn.utils.rnn.pack_padded_sequence(arr_input, arr_seq_len, batch_first=True, enforce_sorted=False)

print(f"{pack.data.shape}, {pack.batch_sizes.shape}")

out, h = gruLayer(pack)
print(f"{out.data.shape}, {h.shape}")
</code></pre>
<p>With the following outputs:</p>
<pre><code class="lang-auto">torch.Size([256, 90, 4]), torch.Size([256])
torch.Size([10179, 4]), torch.Size([89])
torch.Size([10179, 32]), torch.Size([2, 256, 32])
</code></pre>
<p>I don’t understand why the output of <code>pack_padded_sequence</code> (data and  batch_sizes):</p>
<p><code>torch.Size([10179, 4]), torch.Size([89])</code></p>
<p>why it’s not data = (90<em>4</em>256) and (batch_sizes = 256)</p>
<p>?</p>
Transformer decoder output ends up in a "trivial solution" for instance segmenttaion

SD1713: <p>Hi everyone,</p>
<p>I am trying an instance segmentation task in the DETR bbox prediction style. I am using a pretrained backbone features as input to a transformer and represent the sequence tokens as the number of instances in my dataset. I am trying to predict instance masks and end up in predicting the same masks for every sample where the predicted mask contains all instances for all samples, instead of the expected one mask per instance.</p>
<ul>
<li>I have multiple instances in a frame.</li>
<li>Ground truth masks in each image are separated (1 unique mask for each instance).</li>
</ul>
<p>I have a softmax in the output, been trying different learning rates, and initialization, but this did not improve the case. Could someone please help as to if they faced similar situations, and how did you overcome this type of error in an instance segmentation task?</p>
<p>Thanks in advance!</p>
"nn.TransformerDecoderLayer" Without Encoder Input

dev1: <p>Hi everyone. I’m trying to implement GPT. For that I have to use decoder only.</p>
<p>However PyTorch Decoder requires Encoder output as “memory” parameter to forward the decoder.</p>
<p>How can I use the PyTorch Decoder without providing input from Encoder for GPT?</p>

nairbv: <p>The naming convention can be a bit confusing. GPT-style models are often referred to as “decoder only” because they require a causal mask not typically used in the encoder side of an encoder-decoder model. Decoders in encoder-decoder models also use cross attention though, which is not found in GPT-style “decoder only” models. It’s the cross attention that’s looking for encoder outputs, and which is giving you trouble. Some might prefer to call GPT-style models “encoder only” instead of “decoder only.”</p>
<p>I haven’t tried it using the PyTorch transformer modules, but your best bet for a GPT-style model might actually be to use the PyTorch encoder instead of decoder. <code>nn.TransformerEncoder.forward</code> does have an <code>is_causal=True</code> parameter. Then again, you may just want to implement your own module for a GPT-style transformer.</p>

dev1: <p>First of all thank you for your answer. I checked the paper again and you seem right. I have another question if its ok.</p>
<p>So BERT is also using nearly the same architecture. Stacked encoders basically. As I can see only the way they got trained (masked and casual LMs) are different. Is that correct?</p>

vdw: <p>Yes, in their core, they are all transformer-based using only “one half”: decoder-only or encoder-only, as <a class="mention" href="/u/nairbv">@nairbv</a> nicely summarized.</p>
<p>The difference is in its uses, i.e., the training setup. GPT-variants utilize an autoregressive training relying in on the “you are not allowed to peak into the future” mask :). BERT is a masked LM where certain words are masked out to get predicted during training.</p>
<p>In short, your understanding is correct.</p>
PyTorch equivalent of np.interpolate

ptuser1: <p>Hi, I’m trying to rewrite some code which uses the np.interpolate function to use a PyTorch equivalent function. Does such a function exist or has anyone written anything similar? Thanks!</p>

ptrblck: <p>It seems the feature request is tracked <a href="https://github.com/pytorch/pytorch/issues/50334">here</a> and it also contains a few implementations you could try to use.</p>
Clone/Copy compiled model

UmarSpa: <p>I have two models with the same architecture that I torch.compile() separately. And then during the first run of the model when the actual data is run through the model, the graphs are built. This process takes some time.</p>
<pre><code class="lang-auto">model_1 = SomeModel()
model_1_compiled = torch.compile(model_1)

model_2 = SomeModel()
model_2_compiled = torch.compile(model_2)


pred_1 = model_1_compiled(input_1)
pred_2 = model_2_compiled(input_2)

</code></pre>
<p>I was wondering if it is possible to do save some time by compiling only one model, and then copy the built graph for the second model?</p>
<p>Something like this:</p>
<pre><code class="lang-auto">model_1 = Model()
model_1_compiled = torch.compile(model_1)

pred_1 = model_1_compiled(input_1)

model_2_compiled = COPY_COMPILED_MODEL(model_1_compiled)

pred_2 = model_1_compiled(input_2)

</code></pre>

marksaroufim: <p>We’re looking at having something sort of like with saving the inductor compilation cache, there’s a POC here <a href="https://discuss.pytorch.org/t/pt-2-0-are-compiled-models-savable/169871/7" class="inline-onebox">PT 2.0 - Are compiled models savable - #7 by marksaroufim</a> and I’ll be working on cleaning it up to ship in time for next release</p>
How to do CUDA traces callback only for uncaught OOM errors?

oren.learn: <p>From this blog post by <a class="mention" href="/u/zdevito">@zdevito</a>, we see that we can add a callback every time it OOMs but this callback happens for caught OOMs and retries. Is there a way to do callback only when it is a uncaught OOM?</p>
<pre><code class="lang-python">def oom_observer(device, alloc, device_alloc, device_free):
    # snapshot right after an OOM happened
    print('saving allocated state during OOM')
    snapshot = torch.cuda.memory._snapshot()
    dump(snapshot, open('oom_snapshot.pickle', 'wb'))

torch._C._cuda_attach_out_of_memory_observer(oom_observer)
</code></pre>
<p>looking at the C++ code, i don’t see an obvious way to add a flag for only when it is uncaught OOMs <a href="https://github.com/pytorch/pytorch/blob/eb03af44eec31ab11b8fcef4607d44a16024062a/torch/csrc/cuda/Module.cpp#L811C36-L811C61" rel="noopener nofollow ugc">cuda/module.cpp</a> <a href="https://github.com/pytorch/pytorch/blob/eb03af44eec31ab11b8fcef4607d44a16024062a/c10/cuda/CUDACachingAllocator.cpp#L1304-L1306" rel="noopener nofollow ugc">CUDACachingAllocator.cpp</a></p>

zdevito: <p>There isn’t an easy way to do it because the callback needs to run when the OOM happens but before the stack unwinds, otherwise a lot of memory will be released before we can take the snapshot. It is possible to record multiple snapshots and then only save the last one. Each snapshot save will take some time but it will eventually get the actual OOM snapshot:</p>
<pre><code class="lang-auto">def oom_observer(device, alloc, device_alloc, device_free):
    # snapshot right after an OOM happened
    global snapshot
    snapshot = torch.cuda.memory._snapshot()
    
torch._C._cuda_attach_out_of_memory_observer(oom_observer)
try:
  &lt;application code&gt;
except torch.cuda.OutOfMemoryError:
  print('saving allocated state during OOM')  
  dump(snapshot, open('oom_snapshot.pickle', 'wb'))
</code></pre>
Rationale behind not being able to use "arg" as a keyword parameter for nn.Sequential?

NeuralPirate: <p>The <a href="https://pytorch.org/docs/stable/generated/torch.nn.Sequential.html" rel="noopener nofollow ugc">documentation for nn.Sequential</a> mentions the following signature: <code>torch.nn.Sequential(*arg: OrderedDict[str, Module])</code>.</p>
<p>However:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; from collections import OrderedDict
&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.nn.Sequential(arg=OrderedDict({"foo": torch.nn.Identity()}))
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
TypeError: Sequential.__init__() got an unexpected keyword argument 'arg'
</code></pre>
<p>If <code>nn.Sequential</code> was a regular Python class I could use <code>arg</code> as a keyword or a positional argument but it seems that is not the case here. I’m wondering if there is a specific rationale behind this API or if it is just “nobody did invest time to implement this”? Do you think it is worthwhile opening an issue in the github tracker?</p>
<p>Background: The current state makes it harder for me to instantiate a <code>nn.Sequential</code> instance using hydra + yaml.</p>
Is there a version of the rotary embedding with unpadded inputs

Thibault712: <aside class="onebox githubblob" data-onebox-src="https://github.com/HazyResearch/flash-attention/blob/main/flash_attn/layers/rotary.py">
  <header class="source">

      <a href="https://github.com/HazyResearch/flash-attention/blob/main/flash_attn/layers/rotary.py" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/HazyResearch/flash-attention/blob/main/flash_attn/layers/rotary.py" target="_blank" rel="noopener nofollow ugc">HazyResearch/flash-attention/blob/main/flash_attn/layers/rotary.py</a></h4>


      <pre><code class="lang-py"># Copyright (c) 2023, Tri Dao.

from typing import Tuple
import math

import torch

from einops import rearrange, repeat

import rotary_emb


def rotate_half(x, interleaved=False):
    if not interleaved:
        x1, x2 = x.chunk(2, dim=-1)
        return torch.cat((-x2, x1), dim=-1)
    else:
        x1, x2 = x[..., ::2], x[..., 1::2]
        return rearrange(torch.stack((-x2, x1), dim=-1), '... d two -&gt; ... (d two)', two=2)

</code></pre>



  This file has been truncated. <a href="https://github.com/HazyResearch/flash-attention/blob/main/flash_attn/layers/rotary.py" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>I can see that this one working with flash attention is only for padded inputs.</p>
Dataset half precision

caesar025: <p>As far as I understand, using PyTorchs native AMP performs (part) of the <strong>model training</strong> in half precision. Does it at the same time also make sense to force the <strong>dataset</strong> (e.g. ImageNet) to be in half-precision for additional speed-ups or is that already happening automatically as well when using AMP?</p>

ptrblck: <p>Native AMP’s <code>autocast</code> will transform data to a lower-precision <code>dtype</code> (<code>float16</code> or <code>bfloat16</code> depending on the used setup) for eligible operations for you. There is no need to manually transform data or any input to this <code>dtype</code>.</p>
Could upstream support torch1.13.x+cu118

wengyao04: <p>Hi, we want to run our training application on H100 GPUs. We have tested a simple model using torch2.0.1+cu11.8 on H100 which works fine. However, we are still in the process of upgrading torch from 1.x to 2.x. Some LLM models are still in 1.13.x. In order to train these models on H100 GPUs, we need torch1.13.x+cu118 which is not supported yet. Could torch upstream support torch1.13.x+cu118 wheels ?</p>
<p>We also want to upgrade to torch2.1.0+cu121 in the future, do we have a timeline for support it in a stable version, currently, there are only nightly built wheels.</p>
<p>Thanks !</p>

ptrblck: <aside class="quote no-group" data-username="wengyao04" data-post="1" data-topic="183975">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wengyao04/48/50841_2.png" class="avatar"> wengyao04:</div>
<blockquote>
<p>Could torch upstream support torch1.13.x+cu118 wheels ?</p>
</blockquote>
</aside>
<p>No, we won’t rebuilt and publish already released PyTorch versions so you would need to build <code>1.13.x</code> from source using CUDA 11.8.</p>
<aside class="quote no-group" data-username="wengyao04" data-post="1" data-topic="183975">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wengyao04/48/50841_2.png" class="avatar"> wengyao04:</div>
<blockquote>
<p>We also want to upgrade to torch2.1.0+cu121 in the future, do we have a timeline for support it in a stable version, currently, there are only nightly built wheels.</p>
</blockquote>
</aside>
<p>The next release, PyTorch <code>2.1.0</code>, is targeting binaries with CUDA 11.8 and 12.1.</p>
Could not detect GPU when running from docker

laro: <p>I have Ubuntu 20.04.6 LTS with CUDA 11.4 and cuDNN 8500, python 3.8.10.<br>
When running the following command (not from docker):</p>
<pre><code class="lang-auto">import torch
print(torch.__version__)
print(torch.backends.cudnn.version())
print(torch.cuda.is_available())
</code></pre>
<p>I’m getting:</p>
<pre><code class="lang-auto">1.13.1+cu117
8500
True
</code></pre>
<p>I build docker with the following dockerfile:</p>
<pre><code class="lang-auto">FROM python:3.8.10

RUN apt-get update &amp;&amp; apt-get upgrade -y

RUN python -m venv /venv
ENV PATH="/venv/bin:$PATH"

RUN pip install --upgrade pip
RUN pip install torch==1.13.1+cu117 -f https://download.pytorch.org/whl/torch_stable.html
RUN pip install notebook

EXPOSE 8873

WORKDIR /app

COPY . /app

CMD ["jupyter", "notebook", "--ip=0.0.0.0", "--port=8873", "--no-browser", "--allow-root"]
</code></pre>
<p>I run the docker with the followinng command:<br>
<code>sudo docker run -p 8873:8873 myimagenew</code></p>
<p>And run the same code:</p>
<pre><code class="lang-auto">import torch
print(torch.__version__)
print(torch.backends.cudnn.version())
print(torch.cuda.is_available())
</code></pre>
<p>But there the GPU is not detected (output):</p>
<pre><code class="lang-auto">1.13.1+cu117
8500
False
</code></pre>
<p>What is missing and how can I build docker with python 3.8.10 and pytorch which use the GPU ?</p>

Cyber_punk: <p>You’ve missed the argument to allow docker container to access GPU. Try:</p>
<blockquote>
<p>sudo docker run -p 8873:8873 --gpus all myimagenew</p>
</blockquote>
<p>See more: <a href="https://docs.docker.com/engine/reference/commandline/run/#gpus" class="inline-onebox" rel="noopener nofollow ugc">docker run | Docker Docs</a></p>