akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
CUDA installation

username: Caio_Nogueira
cooked: <p>I installed <code>Pytorch</code> and <code>cudatoolkit</code> in an Anaconda environment via the instructions provided in the official documentation <a href="https://pytorch.org/get-started/locally/" rel="noopener nofollow ugc">here</a>. However the output to <code>torch.cuda.is_available()</code> is still False. However, Pytorch seems to be detecting CUDA correctly as it outputs<br>
the correct version (11.8) when I prompt <code>python -m torch.cuda.version</code>.</p>
<p>The Python version installed in the environment is 3.10.11, which I suspect to be the cause of this issue.</p>

username: ptrblck
cooked: <p>Assuming the correct PyTorch binary was installed the output could indicate a driver issue and you might need to reinstall the NVIDIA driver.</p>
How to visualize detection pretrained models layer by layer?

username: MOHAND99
cooked: <p>the most appropriate soln I found was this one<br>
<a href="https://pytorch.org/vision/stable/feature_extraction.html" class="inline-onebox" rel="noopener nofollow ugc">Feature extraction for model inspection — Torchvision main documentation</a> but i end up w/ this error</p>
<p><code>TraceError: Proxy object cannot be iterated. This can be attempted when the Proxy is used in a loop or as a *args or **kwargs function argument. See the torch.fx docs on pytorch.org for a more detailed explanation of what types of control flow can be traced, and check out the Proxy docstring for help troubleshooting Proxy iteration errors</code></p>
<p><strong>My Code</strong></p>
<pre><code class="lang-auto">from torchvision.models.feature_extraction import create_feature_extractor
from torchvision.models.detection import fasterrcnn_mobilenet_v3_large_320_fpn

m = fasterrcnn_mobilenet_v3_large_320_fpn()

return_nodes = {
    'layer1': 'layer1'
}

create_feature_extractor(m, return_nodes=return_nodes)
return_nodes</code></pre>

username: giorgiocoal
cooked: <p>HI!<br>
Same problem here, did you ever find a solution?</p>
RuntimeError: permute(sparse_coo): number of dimensions in the tensor input does not match the length of the desired ordering of dimensions

username: farihachaiti
cooked: <p>So, I am using this clip model for some labelling task. But when I use the clip model’s text encoder, it gives the following error:</p>
<pre><code class="lang-auto"> &lt;ipython-input-117-4c513cc2d787&gt; in forward(self, batch)
     34     print(y.size())
     35     print(y.dim())
---&gt; 36     y = self.text_encoder(y)
     37     y = self.classifier(y)
     38 

/usr/local/lib/python3.10/dist-packages/clip/model.py in encode_text(self, text)
    345         x = x + self.positional_embedding.type(self.dtype)
    346         x = x.permute(1, 0, 2)  # NLD -&gt; LND
--&gt; 347         x = self.transformer(x)
    348         x = x.permute(1, 0, 2)  # LND -&gt; NLD
    349         x = self.ln_final(x).type(self.dtype)

RuntimeError: permute(sparse_coo): number of dimensions in the tensor input does not match the length of the desired ordering of dimensions i.e. input.dim() = 4 is not equal to len(dims) = 3
</code></pre>
<p>The thing is, the labels are multiple for one image, so I am using a collate_fn with pad_sequence in the dataloader before feeding into the model.</p>
<pre><code class="lang-auto"> def pad_sequence(batch):
    return torch.nn.utils.rnn.pad_sequence(batch, batch_first=True, padding_value=0)
  def my_collate_fn(batch):    
    batch['i'] = torch.stack(batch['i'].float())
    batch['l'] = pad_sequence(batch['l'].long())      
    return batch

 class CustomCLIP(torch.nn.Module):
  def __init__(self, num_classes: int = 10, bias=False):
    super().__init__()
    #model, _ = clip.load("RN50")

  def forward(self, batch): 
    x = batch['i']
    x = self.encoder(x)
    x = self.classifier(x)
    y = batch['l']
    print(y)
    print(y.size())
    print(y.dim())
    y = self.text_encoder(y) #error on this line
    y = self.classifier(y)
    x_similarity = x @ x.T
    y_similarity = y @ y.T
    targets = F.softmax(
      (x_similarity + y_similarity) / 2 * temperature, dim=-1
    )
    outputs = (y @ x.T) / temperature

    return outputs, targets
</code></pre>
<p>I have printed out the dimensions of <code>y</code> . its 3 which matches the length dimension. then why is it giving error that the input tensor dimension is 4?</p>
<pre><code class="lang-auto"> [[49406,   332, 49407,  ...,     0,     0,     0],
         [49406,   320, 49407,  ...,     0,     0,     0],
         [49406,   333, 49407,  ...,     0,     0,     0],
         ...,
         [    0,     0,     0,  ...,     0,     0,     0],
         [    0,     0,     0,  ...,     0,     0,     0],
         [    0,     0,     0,  ...,     0,     0,     0]]], device='cuda:0')
torch.Size([32, 392, 77])
3
</code></pre>
<p>Someone please point out whats the issue and how to solve it. Thanks in advance.</p>
How can I change Tensorflow's 'SAME' padding in Pytorch?

username: ymin2570
cooked: <p>I have Tensorflow code as follow:</p>
<pre><code class="lang-auto">kernel_x = tf.reshape(tf.constant([[0, 0], [-1, 1]], tf.float32), [2, 2, 1, 1])
gradient_orig = tf.nn.conv2d(input_tensor, kernel_x, strides=[1, 1, 1, 1], padding='SAME')
</code></pre>
<p>I’ll change this code for Pytorch but I have some trouble.</p>
<pre><code class="lang-auto">import torch
import torch.nn.functional as F

kernel_x = torch.tensor([[0, 0], [-1, 1]], dtype=torch.float32).view(1, 1, 2, 2)
gradient_orig = F.conv2d(input_tensor, kernel_x, stride=1, padding= ??? )
</code></pre>
<p>how can I do the same operation with Tensorflow’s ‘SAME’ padding in this setting?</p>

username: ptrblck
cooked: <p>Did you try to pass <code>padding="same"</code>?</p>

username: ymin2570
cooked: <p>I tried, but I think I got the different results.</p>
<pre><code class="lang-auto">import tensorflow as tf

kernel = tf.reshape(tf.constant([[0, 0], [-1, 1]], tf.float32), [2, 2, 1, 1])
input_tensor = tf.reshape(tf.constant([[1,1],[1, 2]], tf.float32), [2, 2, 1, 1])
gradient_orig = tf.nn.conv2d(input_tensor, kernel, strides=[1, 1, 1, 1], padding='SAME')
print(gradient_orig)
</code></pre>
<pre><code class="lang-auto">tf.Tensor(
[[[[-1.]]

  [[ 0.]]]


 [[[-2.]]

  [[ 0.]]]], shape=(2, 2, 1, 1), dtype=float32)
</code></pre>
<pre><code class="lang-python">import torch
import torch.nn.functional as F

kernel_x = torch.tensor([[0, 0], [-1, 1]], dtype=torch.float32).view(1, 1, 2, 2)
input_tensor = torch.tensor([[1,1],[1,2]], dtype=torch.float32).view(1,1,2,2)
gradient_orig = F.conv2d(input_tensor, kernel_x, stride=1, padding="same")
print(gradient_orig)
</code></pre>
<pre><code class="lang-auto">tensor([[[[ 1., -2.],
          [ 0.,  0.]]]])
</code></pre>
<p>I implement both simple code for the test, but results seem different.<br>
and I got Warning message :<br>
“UserWarning: Using padding=‘same’ with even kernel lengths and odd dilation may require a zero-padded copy of the input be created (Triggered internally at  …\aten\src\ATen\native\Convolution.cpp:647.)”</p>
<p>I’m sorry that I don’t know about tensorflow’s tensor shape… maybe the difficulty came from this problem</p>

username: ptrblck
cooked: <p>I think you might be mixing shapes, as e.g. in TF it seems your input has 2 samples.<br>
From the <a href="https://www.tensorflow.org/api_docs/python/tf/keras/layers/Conv2D">TF/Keras docs</a>:</p>
<blockquote>
<p>Input shape<br>
4+D tensor with shape: <code>batch_shape + (channels, rows, cols)</code> if <code>data_format='channels_first'</code> or 4+D tensor with shape: <code>batch_shape + (rows, cols, channels)</code> if <code>data_format='channels_last'</code>.</p>
</blockquote>
<p>In both memory formats the first dimension should represent the batch dimension, which is set to <code>2</code> in your example while it’s <code>1</code> in PyTorch.</p>
How to train overlapping model parts with different datasets

username: Ethan_Lee
cooked: <p>Hello everyone…! I’ve been searching all around the internet, but I’m not sure how to implement this situation. I am stuck with this problem for weeks and so confused.</p>
<p>Assume a model with partA and partB.<br>
And we have data1 and data2.</p>
<p>My goal is to train partA and partB with data1 and train partB with data2 simultaneously.</p>
<p>This is a pseudo code for what I implemented in pytorch.</p>
<pre><code class="lang-auto">def train():
    model.train()
    for input, target in data1:
        loss1=criterion(model(input),target)
        optimizer.zero_grad()
        loss1.backward()
        
        input, target in data2:
        model.partA.requires_grad=False
        loss2=criterion(model(input),target)
        loss2.backward()
        model.partA.requires_grad=True
        
        optimizer.step()
</code></pre>
<p>The iteration with data1 and data2 is simultaneously repeated.</p>
<p>Is my implementation reasonable? IF NOT, can you guys give me an advice about training a model’s separate parts?</p>
<p>Thx for all of you.</p>

username: J_Johnson
cooked: <p>The problem I see in the above pseudocode is you’re trying to use the same optimizer and then calling <code>.zero_grad()</code> on it between data1 and data2. That means your gradients from data1 are getting wiped before you call <code>.step()</code>.</p>

username: Ethan_Lee
cooked: <p>Oh, my mistake.</p>
<p>I accidentally wrote another ‘optimizer.zero_grad’ during the copy of code.</p>
<p>I edited the pseudocode.</p>
<p>Thanks.</p>

username: J_Johnson
cooked: <p>Here is a sample code you can run to see what happens:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

batch_size=10

model = nn.Sequential(nn.Linear(2,5), nn.Linear(5,1))

optimizer = torch.optim.Adam(model.parameters(), lr=0.001)

criterion = nn.MSELoss()

data1 = torch.rand(batch_size, 2)
data2 = torch.rand(batch_size, 2)

targ1 = torch.rand(batch_size, 1)
targ2 = torch.rand(batch_size, 1)

loss1 = criterion(model(data1), targ1)
optimizer.zero_grad()
loss1.backward()
for param in model.parameters():
    print(param.grad)
model[0].requires_grad_(False)
loss2 = criterion(model(data2), targ2)
loss2.backward()
model[0].requires_grad_(True)
for param in model.parameters():
    print(param.grad)

optimizer.step()
</code></pre>
<p>As expected, the first Linear layer gradients do not change during the entry of data2, yet the second layer does. Hope it helps.</p>

username: Ethan_Lee
cooked: <p>Thank you very much.</p>
<p>Since I am a newbie in pytorch, this helped me a lot.</p>
Form a 3D tensor from 2D using uneven padding with stride size

username: tanyaisanumber
cooked: <p>Given a 2D tensor <code>A</code> of size <code>m x r</code>, and a list of steps <code>steps = [m_1, ..., m_k]</code> that partition <code>m</code> (ie, <code>sum(steps)=m</code>).<br>
Define <code>max_m_i = \max_i  m_i</code> and <code>s_i = sum(steps[:i])</code>.<br>
I need to create a 3D tensor of size <code>k x max_m_i x r</code> by padding every slice of <code>A</code> w.r.t. steps, ie, <code>A[s_i:s_i+steps[i], :]</code> of size <code>m_{i+1} x r</code> pad to the size <code>max_m_i x r</code>.</p>
<p>Example,</p>
<pre><code class="lang-auto">import torch

m = 6; r = 2
A = torch.arange(0, 12).reshape(6, 2)
print("A = ", A)
steps = [3, 2, 1]
print(f"{steps = }")
max_m_i = max(steps)
pad_A = []
s_i = 0
for i in range(len(steps)):
    pad_A += [torch.cat((A[s_i:s_i + steps[i]], torch.zeros((max_m_i - steps[i], r))), axis=0)]
    s_i += steps[i]
pad_A = torch.stack(pad_A)
print("pad_A = ", pad_A)
</code></pre>
<p>output</p>
<pre><code class="lang-auto">A =  tensor([[ 0,  1],
        [ 2,  3],
        [ 4,  5],
        [ 6,  7],
        [ 8,  9],
        [10, 11]])
steps = [3, 2, 1]
pad_A =  tensor([[[ 0.,  1.],
         [ 2.,  3.],
         [ 4.,  5.]],

        [[ 6.,  7.],
         [ 8.,  9.],
         [ 0.,  0.]],

        [[10., 11.],
         [ 0.,  0.],
         [ 0.,  0.]]])
</code></pre>

username: ptrblck
cooked: <p>Creating a mask tensor might give you a performance benefit, but you might need to profile both approaches:</p>
<pre><code class="lang-python">out = torch.zeros(len(steps), max(steps), A.size(1)).long()
mask = torch.tensor(False).expand(len(steps), max(steps), A.size(1)).contiguous()
for s, m in zip(steps, mask):
    m[:s] = True

out[mask] = A.view(-1)
print(out)
# tensor([[[ 0,  1],
#          [ 2,  3],
#          [ 4,  5]],

#         [[ 6,  7],
#          [ 8,  9],
#          [ 0,  0]],

#         [[10, 11],
#          [ 0,  0],
#          [ 0,  0]]])
</code></pre>
<p>I guess there might be a more efficient way, but I didn’t find one quickly.</p>
Does torch.unbind() create a view or a copy?

username: Sam_Lerman
cooked: <p>Does <code>torch.unbind()</code> create a tuple of views or copes of the original tensor i.e. is new memory allocated?</p>
<p>What about when applied to <code>shared_memory</code> tensors?</p>
<p>Is there a way to re-bind them?</p>

username: ptrblck
cooked: <p>Based on this code snippet it’s a view:</p>
<pre><code class="lang-python">x = torch.randn(3, 10)
y = torch.unbind(x, 0)
y[0].zero_()
print(x)
# tensor([[ 0.0000e+00,  0.0000e+00,  0.0000e+00,  0.0000e+00,  0.0000e+00,
#           0.0000e+00,  0.0000e+00,  0.0000e+00,  0.0000e+00,  0.0000e+00],
#         [ 1.1271e+00, -1.6113e-01, -9.3410e-01,  1.2211e-01,  8.4843e-01,
#           4.7476e-01, -8.4909e-01, -1.1142e-01, -4.4881e-01, -3.7674e-01],
#         [-1.5271e+00, -1.0888e+00, -1.3062e-03, -2.3746e-01, -1.6353e+00,
#           1.0766e+00,  4.8345e-01, -6.0865e-01,  8.8547e-02,  3.6673e-01]])
</code></pre>
Training an autoregressive RNN

username: Messiah
cooked: <p>I want to train an autoregressive RNN as follows:</p>
<pre><code class="lang-auto">for i in range(max_seq_len):
    pred, hidden = GRU(pred, hidden) # pred.requires_grad = True
    tot_timestep_loss += loss_fn(pred, target[:, i])
tot_timestep_loss.backward()
optim.step()
</code></pre>
<p>If I want to backpropogate through the output-to-input autoregressive connection via <code>pred</code>, would the above be sufficient? Specifically, I want to ensure the autoregressive mapping is considered in the evaluation of the gradient during backprop such that in theory the model parameters are influenced by the free-running autoregressive decoding errors during training. When I print the gradient of the non-leaf variable <code>pred</code> using <code>pred.register_hook()</code>, I notice it returns a tensor of size <code>BxD</code>, where <code>B</code> is the batch size and <code>D</code> is the output dimension, which I think is associated with the final timestep gradient at <code>max_seq_len</code>. Is the model accurately accounting for the autoregressive term in the gradient calculation using this method? Is there an easy way to visualize the backprop graph to see if these connections exist?</p>
Where is bmm source code? When using Inductor backend, it is not generating Triton kernels?

username: tianxiangsong
cooked: <p>TorchDynamo supports many different backends but inductor specifically works by generating <a href="https://github.com/openai/triton" rel="noopener nofollow ugc">Triton</a> kernels and we can inspect them by running <code>TORCH_COMPILE_DEBUG=1 python trig.py</code>.<br>
But when using example can not get matmul kernel.  How compute it and where can I get source code?</p>
<pre><code class="lang-auto">import torch
def fn(x, y):
    return torch.matmul(x, y).cuda()
new_fn = torch.compile(fn, backend="inductor")
input_tensor = torch.randn(768,768).to(device="cuda:0")
a = new_fn(input_tensor, input_tensor)
print(a)
</code></pre>

username: eqy
cooked: <p>You can inspect the names of kernels that are executed via e.g., <code>nsys nvprof</code>, though I would suspect for a compute-intensive kernel like BGEMM it would be dispached to cuBLAS: <a href="https://github.com/pytorch/pytorch/blob/fb468b6792213e0d8e6221b3bb51e71fcadbed30/aten/src/ATen/cuda/CUDABlas.cpp#L233" class="inline-onebox" rel="noopener nofollow ugc">pytorch/CUDABlas.cpp at fb468b6792213e0d8e6221b3bb51e71fcadbed30 · pytorch/pytorch · GitHub</a></p>
<p>cuBLAS kernels in general are not open-source, though you might some implementations that are comparable in performance in CUTLASS: <a href="https://github.com/NVIDIA/cutlass" class="inline-onebox" rel="noopener nofollow ugc">GitHub - NVIDIA/cutlass: CUDA Templates for Linear Algebra Subroutines</a></p>
Slightly different results in same machine and GPU but different order

username: cgr71ii
cooked: <p>Hi!</p>
<p>I’ve noticed that when I provide different order of my data at inference-time, the results are slightly different. I’ve put <code>model.eval()</code>, and everything is fine if the order is the same, but I tried to <code>shuf</code> since I was observing different results in the <code>flask</code> server I was using, then I noticed this problem. Is this expected?</p>
<p>Example:</p>
<pre><code class="lang-bash">cat data | wc -l # 300.000

for i in $(seq 1 100); do
  cat data | inference | md5sum # Always same MD5
done

cat data | shuf | inference | md5sum # Different MD5

comm -3 &lt;(cat data | inference | sort) &lt;(cat data | shuf | inference | sort) | wc -l # 6 -&gt; 3 different results
</code></pre>
<p>I’ve checked out similar posts like:<br>
<a href="https://discuss.pytorch.org/t/slightly-different-results-when-evaluating-same-model-on-different-machines">https://discuss.pytorch.org/t/slightly-different-results-when-evaluating-same-model-on-different-machines</a><br>
<a href="https://discuss.pytorch.org/t/slightly-different-results-on-k-40-v-s-titan-x">https://discuss.pytorch.org/t/slightly-different-results-on-k-40-v-s-titan-x</a></p>
<p>but they don’t describe exactly the same situation.</p>
<p>GPU: GeForce RTX 2080 Ti<br>
NVIDIA driver version: 525.85.12<br>
CUDA: 12.0</p>
<p>Thank you!</p>

username: matt.carlson
cooked: <p>Which type of model architecture are you using? I have seen a lot of transformers that take in a seed as a parameter suggesting that there are variable results at inference time. I don’t have a concrete answer though, just an observation.</p>

username: cgr71ii
cooked: <p>XLM-RoBERTa from HuggingFace. But when I provide the inference data in the same order multiple times, as can be observed in the pseudo execution I provided, the result is the same. The thing is that when the order is different, the results are slightly different. Anyway, do you think this could be related to the architecture?</p>

username: matt.carlson
cooked: <p>I do. Some (most?) models are non-deterministic. if you run the inference twice through the loop without instantiating a new model are they identical?</p>
Torch multinomial in generate function

username: mohit8
cooked: <p>Hi, I came across a generate code in the language model that is as follows :-</p>
<pre><code class="lang-auto">    def generate(self, idx, max_new_tokens):
        for _ in range(max_new_tokens):
            idx_cond = idx[:, -block_size:]
            logits, loss = self(idx_cond)
            logits = logits[:, -1, :]
            probs = F.softmax(logits, dim = -1)
            id_next = torch.multinomial(probs, num_samples = 1)
            idx = torch.cat((idx, id_next), dim = 1)
        return idx
</code></pre>
<p>I have two questions regarding it</p>
<p>Q1) Why is torch.multinomial used instead of torch.argmax for selecting the index of the next token to generate?</p>
<p>Q2) Why are we using torch.multinomial only why not some other probability distribution?</p>

username: KFrank
cooked: <p>Hi Mohit!</p>
<p>In regard to your questions about <code>multinomial()</code> (but not trying to comment<br>
on the larger use case that <code>generate()</code> is presumably a part):</p>
<aside class="quote no-group" data-username="mohit8" data-post="1" data-topic="180682" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mohit8/48/45534_2.png" class="avatar"> mohit8:</div>
<blockquote>
<p>Q1) Why is torch.multinomial used instead of torch.argmax for selecting the index of the next token to generate?</p>
</blockquote>
</aside>
<p>Let’s say that your two largest <code>probs</code> are rather close together (for example,<br>
<code>0.25</code> and <code>0.26</code>).  Using <code>argmax()</code> would always give you the index of <code>0.26</code>,<br>
ignoring, in a sense, that <code>0.25</code> is almost the same.  On the other hand, using<br>
<code>multinomial()</code> will give you the index of <code>0.26</code> 26% of the time and the index<br>
of <code>0.25</code> 25% of the time, respecting the fact that the two values are quite close<br>
to one another.  (This may or may not be the behavior you want, but it does<br>
make sense.)</p>
<aside class="quote no-group">
<blockquote>
<p>Q2) Why are we using torch.multinomial only why not some other probability distribution?</p>
</blockquote>
</aside>
<p>You could use other probability distributions and something else (like using<br>
<code>argmax()</code>, which is the probability distribution that gives you one specific index<br>
100% of the time) might better fit your use case.  However, as noted above,<br>
using <code>multinomial()</code> samples index values according to <code>probs</code>, which could<br>
well be what you want.</p>
<p>Best.</p>
<p>K. Frank</p>
VRAM usage increase with more gradient accumulation steps

username: ipoletaev
cooked: <p>What would cause GPU memory usage increase with more gradient accumulation steps? Essentially inside the following loop</p>
<pre><code class="lang-auto"># outer loop
	optimizer.zero_grad(set_to_none=True)
	for micro_step in range(gradient_accumulation_steps):
	    batch = next(train_data_loader_iterator)
	    no_sync_enabled = micro_step &lt; gradient_accumulation_steps - 1
	    with no_backward_sync_ctx(enabled=no_sync_enabled): # no-op for a single device
	        loss = model(**batch)
	        loss /= gradient_accumulation_steps
	        loss.backward(loss)
	        total_loss += loss.detach()
	optimizer.step()
</code></pre>
<p>I am assuming changing <code>gradient_accumulation_steps</code> <em>shall not</em> have any impact on the memory usage. Gradient accumulation shouldn’t create any new variables and all the parameter-specific updates should be accumulated in-place (e.g. <code>update += new_update</code>)?</p>
<p>In practice, however, while training the model on a single GPU, multiple GPUs with DDP, or FSDP - everywhere I observe higher VRAM usage for number of steps more than 1.  Any idea whether it is expected? And if not - how to debug this “leak”?</p>
<p>Thanks!</p>

username: eqy
cooked: <p>In theory, this is expected behavior if we consider that peak memory usage in a typical forward, then backward model execution occurs just before the backward pass as at this stage forward activations are being kept alive in preparation for the backward pass.</p>
<p>At this point, in the regime without gradient accumulation (<code>gradient_accumulation_steps=1</code>), the model parameters do not have gradient tensors because <code>optimizer.zero_grad(set_to_none=True)</code> was called, so the gradient tensors are <code>None</code> rather than only being zero’d out. However, with <code>gradient_accumulation_steps&gt;1</code>, these gradient tensors are kept around at this stage for, well, gradient accumulation so the high watermark of memory usage would be higher.</p>
<p>In other words, for the first iteration of <code>for micro_step in range(gradient_accumulation_steps):</code>, the gradient fields of the parameters are not populated yet, but for later iterations, they will be, increasing the maximum total memory usage at this point. The <em>very first</em> accumulation is not in-place in this sense.</p>
<p>However, if you are observing a trend of increasing memory usage when increasing the number of gradient accumulation steps beyond 2, then this could be unexpected and indicative of a leak.</p>
Weight in a module not showing up in named_parameters?

username: Goldname
cooked: <p>I pruned a simple model and when I try to show all the named_parameters, “weight” and “weight_mask” don’t doesn’t appear despite them existing (as you can see from the last two print statements). Why is this?</p>
<pre><code class="lang-auto">import torch.nn as nn
import torch.nn.functional as F
import torch
import torch.nn.utils.prune as prune

class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(2,5)
        self.fc3 = nn.Linear(5, 2)

    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = self.fc3(x)
        return x
net = Net()

prune.l1_unstructured(net.fc3, name="weight", amount = 1)
for name, param in net.fc3.named_parameters():
    print(name)
# The above only prints weight_orig. It should also print "weight" and "weight_mask"

print(net.fc3.weight)
print(net.fc3.weight_mask)
</code></pre>

username: mikaylagawarecki
cooked: <p>Hey <a class="mention" href="/u/goldname">@Goldname</a>, looking at the <a href="https://pytorch.org/docs/stable/generated/torch.nn.utils.prune.l1_unstructured.html#torch.nn.utils.prune.l1_unstructured" rel="noopener nofollow ugc">documentation for <code>torch.nn.utils.prune.l1_unstructured</code></a> <code>weight_mask</code> should be stored in <code>named_buffers</code>.</p>
<p>With respect to missing <code>weight</code> that seems inconsistent with the documentation. It might be a bug perhaps you should <a href="https://github.com/pytorch/pytorch/issues/new/choose" rel="noopener nofollow ugc">file an issue</a>.</p>
How to use slices with an accessor in libtorch?

username: velenos14
cooked: <p>I want to update a 2-dimensional slice of a 3D tensor in libtorch.</p>
<p>I have made an accessor for this 3D tensor.</p>
<ol>
<li>
<p>Would it be quicker to use the accessor to populate the 2D slice? Rather than doing what I was doing up until now, i.e.: <code>Psi_spec.index_put_( {i, Slice(), Slice()}, zero_imag_tensor );</code></p>
</li>
<li>
<p>If this is the case, how shall use the accessor to obtain what I obtain with the code above at 1. ?<br>
<code>Psi_spec_acc.index_put_( {i, Slice(), Slice()}, zero_imag_tensor );</code> fails.<br>
Similarly, <code>Psi_spec_acc[i] = zero_imag_tensor;</code> fails as well.<br>
Moreover, <code>Psi_spec_acc[i][][] = zero_imag_tensor;</code> fails again.</p>
</li>
</ol>
<p>For completeness:<br>
<code>torch::Tensor zero_imag_tensor = torch::zeros( {1}, torch::TensorOptions().dtype(torch::kComplexDouble) );</code></p>
Is there torch::compile in C++?

username: bitnick10
cooked: <p>python version of torch.compile is cool! Is there C++ version of compile?</p>

username: LogicPy
cooked: <p>PyTorch does not have a direct C++ equivalent for torch.jit.compile. The torch.jit namespace in Python is used for PyTorch’s JIT compiler, which takes Python code and compiles it to TorchScript, a statically typed subset of Python that can be optimized and run independently of Python.</p>
<p>For C++, PyTorch provides a TorchScript compiler which can be used to run TorchScript code. This means you can write your model in Python, use torch.jit.script or torch.jit.trace to convert it to TorchScript, save it to a file, and then load that file in C++ and run it.</p>
<p>Here’s a very simplified example:</p>
<p>In Python:</p>
<pre><code class="lang-auto">python
Copy code
import torch
class MyModule(torch.nn.Module):
    def forward(self, x):
        return x * 2

my_module = MyModule()

# Convert to TorchScript via scripting
scripted_module = torch.jit.script(my_module)
# Save to a file
scripted_module.save("my_module.pt")
</code></pre>
<p>Then, in C++:</p>
<pre><code class="lang-auto">cpp
Copy code
#include &lt;torch/script.h&gt;

int main() {
    // Load the TorchScript module
    torch::jit::script::Module module = torch::jit::load("my_module.pt");
    // Use it just like a regular torch::nn::Module
    auto tensor = torch::randn({1});
    std::vector&lt;torch::jit::IValue&gt; inputs;
    inputs.push_back(tensor);
    at::Tensor output = module.forward(inputs).toTensor();
    std::cout &lt;&lt; output &lt;&lt; std::endl;
    return 0;
}
</code></pre>
<p>This enables PyTorch models to be run in a C++ environment. However, please note that the C++ API does not currently support all features of PyTorch, and is intended to be used for deployment of models, rather than for model development.</p>

username: hummlj
cooked: <p>Hi,</p>
<p>pre 2.0, the way to go was to develop in Python, then use TorchScript, save the model, and load it to the C++ Libtorch environment for deployment.</p>
<p>Does that workflow still exist with the new torch.compile (<a href="https://pytorch.org/tutorials/intermediate/torch_compile_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">torch.compile Tutorial — PyTorch Tutorials 2.0.1+cu117 documentation</a>) functionality?</p>
<p>Because I assume the model generated by torch.compile is more optimized (faster, less memory) than a model generated by TorchScript.</p>
<p>Any input?</p>
<p>Best,<br>
J</p>
Fail to install solov2

username: Sylvain_Ard
cooked: <p>Hello,<br>
I don’t succeed to install solov2 with Anaconda on my Windows 10 machine but I really need it.<br>
I post a thread on the solov2 github but they don’t answer : <a href="https://github.com/WXinlong/SOLO/issues/244" class="inline-onebox" rel="noopener nofollow ugc">I have a bug when I launch train.py · Issue #244 · WXinlong/SOLO · GitHub</a><br>
If one of you has succeeded to install it please tell me the commands you typed.<br>
Thank you<br>
Best regards</p>

username: Sylvain_Ard
cooked: <p>no one has an answer please ?</p>

username: ptrblck
cooked: <p>Similar to your previous questions this one also lacks information as well as effort.<br>
E.g. it’s unclear what exactly you’ve tried so far, what exactly is failing, where you are stuck etc.<br>
From your post it seems you expect us to read through this repository and its install instructions ourselves to figure out how to build the package.<br>
However, even if someone would spent this time and got it working, it wouldn’t help as their most likely answer would be that it works for them while it’s still unclear what’s failing for you.</p>

username: Sylvain_Ard
cooked: <p>hello the details of the bug are on the github of solov2, what additional information do you need ?<br>
I post again the link : <a href="https://github.com/WXinlong/SOLO/issues/244" class="inline-onebox" rel="noopener nofollow ugc">I have a bug when I launch train.py · Issue #244 · WXinlong/SOLO · GitHub</a></p>

username: Sylvain_Ard
cooked: <p>I installed it by :</p>
<pre><code class="lang-auto">conda create --name solov2 python=3.7 -y
conda activate solov2
conda install pytorch==1.8.0 torchvision==0.9.0 torchaudio==0.8.0 cudatoolkit=11.1 -c pytorch -c conda-forge
pip install mmdet mmcv==2.0.0
</code></pre>
<p>but I have an error :</p>
<pre><code class="lang-auto">(solov2) C:\Users\MASTER\Desktop\SOLO-master\SOLO-master&gt;python tools/train.py configs/solov2/solov2_light_448_r18_fpn_8gpu_3x.py
Traceback (most recent call last):
  File "tools/train.py", line 9, in &lt;module&gt;
    from mmcv import Config
ImportError: cannot import name 'Config' from 'mmcv' (C:\ProgramData\Anaconda3\envs\solov2\lib\site-packages\mmcv\__init__.py)
</code></pre>
<p>I searched the error on google and they advise to install mmcv-full instead of mmcv but when I install the last version (1.7.1) of mmcv-full I have an error message telling me that I must have mmcv version 2.0</p>

username: Sylvain_Ard
cooked: <p>I finally succeeded. I got the solov2 from the official github which has not be updated since 2020, I got it from mmdetection and succeed</p>

username: Manuel_Alejandro_Dia
cooked: <p>I found your problem.</p>
<p>The repo needs <code>mmcv 0.2.16</code> and you were installing <code>mmcv 2.0.0</code>.</p>
<p>Next time make sure you are installing things properly as indicated in the documentation.</p>

username: Sylvain_Ard
cooked: <p>yes but there are not version of mmdet compatible, the repository is too old</p>
Cannot move a tensor created on the GPU to the CPU in libtorch

username: velenos14
cooked: <p>I cannot move a tensor created on the GPU to the CPU in libtorch.</p>
<p>I do:</p>
<pre><code class="lang-auto">auto options_for_Psi_spec = torch::TensorOptions().dtype(torch::kComplexDouble).device(torch::kCUDA, 0);
torch::Tensor Psi_spec = torch::zeros( {100, 32, 65}, options_for_Psi_spec );
Psi_spec.to(torch::kCPU);
std::cout &lt;&lt; Psi_spec.device() &lt;&lt; std::endl;
</code></pre>
<p>This produces the output:<br>
<code>cuda:0</code></p>
<p>I have also tried with <code>Psi_spec.to("cpu")</code> and with <code>Psi_spec.cpu()</code>, but the same output appears.</p>
<p>What is the command to move a tensor from GPU to CPU in libtorch?</p>

username: ptrblck
cooked: <p>The <code>to()</code> operation is not an inplace operation for tensors so you would need to re-assign the result.</p>
Batch Loading Memory Mapped Image Files Using Tensor DIct

username: Amrit_Das
cooked: <p>Hello, I am working on an experiment, where I want to reduce my training time. For the same, I am using the <span class="mention">@tensorclass</span> and MemoryMapping from<strong>TensorDict</strong>.<br>
I have been following up on the Pytorch Tutorial<br>
<a href="https://pytorch.org/rl/tensordict/tutorials/tensorclass_imagenet.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/rl/tensordict/tutorials/tensorclass_imagenet.html</a></p>
<p>Following is my ImageNetClass</p>
<pre><code class="lang-auto">@tensorclass
class ImageNetData:
    images: torch.Tensor
    targets: torch.Tensor

    @classmethod
    def from_dataset(cls, dataset):
        data = cls(
            images=MemmapTensor(
                len(dataset),
                *dataset[0][0].squeeze().shape,
                dtype=torch.uint8,
            ),
            targets=MemmapTensor(len(dataset), dtype=torch.int64),
            batch_size=[len(dataset)],
        )
        # locks the tensorclass and ensures that is_memmap will return True.
        data.memmap_()

        batch = 64
        dl = DataLoader(dataset, batch_size=batch, num_workers=NUM_WORKERS)
        i = 0
        pbar = tqdm.tqdm(total=len(dataset))
        for image, target in dl:
            _batch = image.shape[0]
            pbar.update(_batch)
            data[i : i + _batch] = cls(
                images=image, targets=target, batch_size=[_batch]
            )
            i += _batch

        return data
</code></pre>
<p>Thats my collate_fn</p>
<pre><code class="lang-auto">class Collate(nn.Module):
    def __init__(self, transform=None, device=None):
        super().__init__()
        self.transform = transform
        self.device = torch.device(device)

    def __call__(self, x: ImageNetData):
        # move data to RAM
        if self.device.type == "cuda":
            out = x.apply(lambda x: x.as_tensor()).pin_memory()
        else:
            out = x.apply(lambda x: x.as_tensor())
        if self.device:
            # move data to gpu
            out = out.to(self.device)
        if self.transform:
            # apply transforms on gpu
            out.images = self.transform(out.images)
        return out
</code></pre>
<p>This is straighforward but the issue I am facing is when itterating through the database using the dataloader<br>
I get this issue as</p>
<pre><code class="lang-auto">Cannot use *apply* to type *list*
</code></pre>
<p>I tried out using List Comprehension and np.vectorize to replicate the apply function but it doesn’t work, if you print the output of the <em><strong>call</strong></em> function, I get batch size and tensor size as empty.</p>
<p>Any help is appreciated.</p>
<p>Pytorch version 1.11<br>
Cuda Version 11.3</p>
Is there a gradgradgrad check? (TripleBackward check)

username: AlphaBetaGamma96
cooked: <p>Hi All,</p>
<p>I’m trying to validate a manually derived derivative up to its third derivative. I’ve been using <code>torch.autograd.gradcheck</code> and <code>torch.autograd.gradgradcheck</code> to check the first and second derivatives respectively (<a href="https://pytorch.org/docs/stable/generated/torch.autograd.gradcheck.html" class="inline-onebox" rel="noopener nofollow ugc">torch.autograd.gradcheck — PyTorch 2.0 documentation</a>). However, there exists no equivalent for the third derivative?</p>
<p>Should I simply just compute the third order derivative and use <code>torch.allclose</code> between the two methods?</p>

username: soulitzer
cooked: <p>You could also have a function that computes the second derivative and pass that to gradcheck.<br>
Or have a function that computes the first derivative and pass that to gradgradcheck.<br>
gradgradcheck itself is just a function that wraps your function to do the first derivative, and then calls into gradcheck underneath</p>

username: AlphaBetaGamma96
cooked: <p><strong>TL;DR</strong> - Is the pseudocode I’ve shown below the correct way to computing a triple-gradcheck? I’ve tried this and my analytical backward is wrong, even though if I compute the triple-jacrev manually and compare with a pytorch primitives version of my custom function is exactly the same. So, I think my <code>jacrev_func</code> function may be wrong.</p>
<p>Hii <a class="mention" href="/u/soulitzer">@soulitzer</a>,</p>
<p>Sorry to reopen this topic, but would it be as simple as applying <code>torch.func.jacrev</code> to my custom function and passing that new function to <code>torch.autograd.gradgradcheck</code>. For example,</p>
<pre><code class="lang-auto">x  = torch.randn(1,)
y = torch.randn(1,)

def func(x,y):
  return myCustomFunction.apply(x,y) #some custom function which has 3 custom derivatives 

def jacrev_func(x,y):
  jacrev_x, jacrev_y = torch.func.jacrev(func, argnums=(0,1))(x,y)
  return jacrev_x, jacrev_y

triple_grad_check = torch.autograd.gradgradcheck(jacrev_func, inputs=(x,y))
</code></pre>
<p>I only ask this as this is what I’ve done, but <code>torch.autograd.gradgradcheck</code> states my derivatives are wrong. I have checked my custom derivatives via calculating a triple-jacrev with my custom function (and comparing it to a pytorch primitives version), and then comparing the two results with an <code>torch.allclose</code> with returns True.</p>
<p>So, there’s a bit of confusing as to why computing the triple-jacrev directly and comparing via <code>torch.allclose</code> returns True, yet <code>torch.autograd.gradgradcheck</code> returns False.</p>
<p>Is the pseudocode I’ve shown above the right way to check the triple-backward?</p>
Configure solov2 in mmdetection

username: Sylvain_Ard
cooked: <p>Hello, I am looking for how to configure solov2 in mmdetection because with my configuration it does not detect any instance.<br>
I posted my question in the official github :</p><aside class="onebox githubissue" data-onebox-src="https://github.com/open-mmlab/mmdetection/issues/10378">
  <header class="source">

      <a href="https://github.com/open-mmlab/mmdetection/issues/10378" target="_blank" rel="noopener nofollow ugc">github.com/open-mmlab/mmdetection</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewbox="0 0 14 16" aria-hidden="true"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/open-mmlab/mmdetection/issues/10378" target="_blank" rel="noopener nofollow ugc">how to parametrize mmdetection for a specific solov2 dataset</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2023-05-23" data-time="15:20:57" data-timezone="UTC">03:20PM - 23 May 23 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/SylvainArd" target="_blank" rel="noopener nofollow ugc">
          <img alt="SylvainArd" src="https://avatars.githubusercontent.com/u/19763527?v=4" class="onebox-avatar-inline" width="20" height="20">
          SylvainArd
        </a>
      </div>
    </div>

    <div class="labels">
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">Hi I want to train my own dataset in COCO format with only one category ("spore"<span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">) with the config file "configs/solov2/solov2_x101-dcn_fpn_ms-3x_coco.py", it is unclear what I must change in the default configuration files. Could you, with this specific configuration, tell me the modifications I must do please ?
Thank you
Kind regards</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
but there are quasi-never answers.<br>
Is some of you has faced the same problem and solved it ?<br>
Thank you<br>
Best regards</p>
Indexing CUDA tensors in LibTorch inside functions

username: velenos14
cooked: <p>Inside a function called from main(), I am trying to iterate over the elements of a CUDA LibTorch tensor in CPP.</p>
<p>I fail with both <code>accessor&lt;double, 3&gt;()</code> and <code>packed_accessor&lt;double, 3&gt;()</code>, similarly with <code>packed_accessor64&lt;double, 3&gt;()</code>.</p>
<p>The accessor is created fine, but when I try to index any element from the underlying tensor, for example:</p>
<pre><code class="lang-auto">auto Psi_spec_abs_acc = Psi_spec_abs.packed_accessor&lt;double, 3&gt;();
std::cout &lt;&lt; Psi_spec_abs_acc [0][0][0] &lt;&lt; std::endl;
</code></pre>
<p>It fails with segfault.</p>
<p>How could I iterate over a CUDA tensor without putting it into the CPU?</p>
<p>Thank you!</p>
Get initialization weights for pre-trained models

username: nipunagarwala
cooked: <p>For some experiments, it seems useful to have access to the initialization weights on pretrained models to do analysis.<br>
For models in torchvision.models , like ResNet-50, is it possible to know what the initialization weights were? If not, could we know what the training recipe/code and seed were to reproduce the initialization weights in local experiments?</p>

username: tiramisuNcustard
cooked: <p><a class="mention" href="/u/nipunagarwala">@nipunagarwala</a>, Please see the following article for methods to get the initialization weights. <a href="https://www.askpython.com/python-modules/initialize-model-weights-pytorch" rel="noopener nofollow ugc">https://www.askpython.com/python-modules/initialize-model-weights-pytorch</a></p>
<p>Summary:</p>
<pre><code class="lang-auto">
`# Linear Dense Layer`
`layer_1 ``=` `nn.Linear(``5``, ``2``)`
`print``(``"Initial Weight of layer 1:"``)`
`print``(layer_1.weight)`

`# Initialization with uniform distribution`
`nn.init.uniform_(layer_1.weight, ``-``1``/``sqrt(``5``), ``1``/``sqrt(``5``))`
`print``(``"\nWeight after sampling from Uniform Distribution:\n"``)`
`print``(layer_1.weight)`
</code></pre>

username: nipunagarwala
cooked: <p>So we do know the function used for initialization, but thats not enough to get the exact weights, is it?<br>
We need to know the seed used for the training process, and whether the seed actually works the same way on different platforms.</p>
<p>Not sure how to exactly replicate it, and if it is even possible.</p>

username: tiramisuNcustard
cooked: <p><a class="mention" href="/u/nipunagarwala">@nipunagarwala</a>, I would like to draw your attention to</p>
<pre><code class="lang-auto">`print``(layer_1.weight)`
</code></pre>
<p><code>in my previous response.</code></p>

username: nipunagarwala
cooked: <p>This is for new layers, not for pretrained models.<br>
Maybe my question wasn’t clear</p>
Parallelization of Graph Neural Network with Temporal Dependencies

username: IZugec
cooked: <p>I am currently working on a research project involving the application of a Graph Neural Network (GNN)-based model for simulations in a scientific domain. Model outputs a matrix based on the input graph. Usually one would then compare this output with true value and compute gradients. In my specific domain, I think, it might be useful to use this output to update the feature of initial graph, which can then again be used as an input to the model (I am using for loop for this, It’s kind of in a spirit of RNN but I am trying to take advantage of graph representation and message passing). Problem is, how do I parallelize this across multiple GPUs as it’s very slow?</p>
<p>Thanks in advance</p>
Is it possible to stack multiple transformations/functions in PyTorch into a single function?

username: jakelevi1996
cooked: <p>Is it possible to stack multiple transformations/functions in PyTorch into a single function? I’m ideally looking for something like this (possibly with more care taken over tensor shapes):</p>
<pre><code class="lang-python">import torch

f_stack = torch.stack([lambda x: x+1, lambda x: x*2, lambda x: x*x])

f_stack(torch.tensor(3))
# &gt;&gt;&gt; torch.Tensor([4, 6, 9])

f_stack(torch.tensor(4))
# &gt;&gt;&gt; torch.Tensor([5, 8, 16])
</code></pre>
<p>(see <a href="https://stackoverflow.com/q/76323287/8477566" rel="noopener nofollow ugc">associated Stack Overflow question</a>)</p>

username: tom
cooked: <p>No, you have to apply the functions and then stack the result.<br>
I recall that in the olden days this would be fused into a single kernel by the fuser if using TorchScript, maybe it still is with torch.compile or so.</p>
<p>Best regards</p>
<p>Thomas</p>

username: jakelevi1996
cooked: <p>Hi <a class="mention" href="/u/tom">@tom</a> , thanks for your reply. Makes sense. I’ll have a small number of functions to apply to different elements from a large batch of data, so I guess the most efficient approach will be for me to use some sort of fancy indexing to extract the sub-tensors I need for each function, and then pass those sub-tensors through the functions separately (vectorising for each function, rather than passing every single element through its own function separately, which would presumably be slower).</p>

username: jakelevi1996
cooked: <p>Just to clarify, the example I gave for what I’m looking for in the original question was a bit unclear, this example is hopefully clearer:</p>
<pre><code class="lang-auto">import torch

f_stack = torch.stack([lambda x: x+1, lambda x: x*2, lambda x: x*x])

f_stack(torch.tensor([0, 0, 0]))
# &gt;&gt;&gt; torch.Tensor([1, 0, 0])

f_stack(torch.tensor([3, 3, 3]))
# &gt;&gt;&gt; torch.Tensor([4, 6, 9])

f_stack(torch.tensor([3, 0, 0]))
# &gt;&gt;&gt; torch.Tensor([4, 0, 0])
</code></pre>
<p>I assume the same answer still stands</p>
Constrain layers' parameters

username: Fabrice_Auzanneau
cooked: <p>Hi<br>
Is it possible to constrain the values of the parameters of a layer to stay in a given range, for example to stay above 0.1 or below -0.1 ?<br>
Thanks for your answers.</p>

username: WANGSSSSSSS
cooked: <p><a href="https://pytorch.org/docs/stable/generated/torch.nn.utils.parametrize.register_parametrization.html#torch.nn.utils.parametrize.register_parametrization" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.utils.parametrize.register_parametrization.html#torch.nn.utils.parametrize.register_parametrization</a></p>

username: WANGSSSSSSS
cooked: <p>register a parametrization function for a certain weight may solve your problem</p>

username: Fabrice_Auzanneau
cooked: <p>Thanks<br>
Actually, what I’m trying to do is the following. I have a linear layer in my model and I want to add somewhere else in the model a new linear layer that would act as the inverse of the previous one.<br>
Say the first linear model does : y = Ax+b<br>
Then, the other layer would compute : y = (X-b)/A, where A and b are the weights and bias tensors of the first linear layer.<br>
From what I understand from parameterization, this is exactly what I need to create the second layer, as a parameterization of the first one.<br>
Do you think this is correct ?</p>