akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Initializing trick in Masked Auto Encoder

Taejune: <p>Hi, currently I’m working with MAE, and got curious about the initializing trick when initialize <code>self.patch_embed</code>.</p>
<p>MAE uses <code>timm.models.vision_transformer</code>’s <code>PatchEmbed</code>, and the <code>PatchEmbed</code> utilizes <code>nn.Conv2d</code> for patchify.</p>
<p>Then the initialization of the <code>PatchEmbed</code>’s conv be like:</p>
<pre><code class="lang-auto"># initialize patch_embed like nn.Linear (instead of nn.Conv2d)
w = self.patch_embed.proj.weight.data
torch.nn.init.xavier_uniform_(w.view([w.shape[0], -1]))
</code></pre>
<p>As written in the comment, weights of the conv are intentionally flatten before its initialization.<br>
So why they make conv weights into the shape of <code>nn.Linear</code>’s?<br>
Is there any advantage of doing so?</p>
<p>Thanks in advance.</p>

ptrblck: <p>Flattening the conv weight will make a difference in the initialization and I guess the authors might see an advantage in their training using it. Did you check the paper to see if they give an explanation there?</p>
<p>Here is a small example showing the difference:</p>
<pre><code class="lang-python">conv = nn.Conv2d(3, 16, 3)

# standard conv weigth shape
fan_in, fan_out = torch.nn.init._calculate_fan_in_and_fan_out(conv.weight)
print(fan_in, fan_out)
# 27 144

gain = 1.0
std = gain * math.sqrt(2.0 / float(fan_in + fan_out))
print(std)
# 0.10814761408717502

a = math.sqrt(3.0) * std 
print(a)
# 0.18731716231633877

# flatten
fan_in, fan_out = torch.nn.init._calculate_fan_in_and_fan_out(conv.weight.view(conv.weight.size(0), -1))
print(fan_in, fan_out)
# 27 16

gain = 1.0
std = gain * math.sqrt(2.0 / float(fan_in + fan_out))
print(std)
# 0.21566554640687682

a = math.sqrt(3.0) * std 
print(a)
# 0.37354368381881414
</code></pre>
<p>Note that <code>a</code> will be used in <code>tensor.uniform_(-a, a)</code> to initialize the tensor.</p>

Taejune: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Hi, Thanks for the reply.<br>
I check the paper and didn’t find any explanation about it.</p>
<p>Therefore I eventually hypothesize that the purpose is to mimic the behavior of patchify that original ViT does.<br>
Even though the original ViT also uses a conv layer in its patchify, the way it actually has to do is to reshape the image into <code>B x N x P^2*C</code>, and apply linear transformation in <code>dim=-1</code>.</p>
<p>Therefore, I think the MAE flattens the conv weight to make the conv layer behave like a linear layer.</p>
<p>Do you think this makes sense?</p>

ptrblck: <p>Yes, I think your explanation sounds reasonable, but let’s also wait for an answer of the authors in the corresponding <a href="https://github.com/facebookresearch/mae/issues/152">issue</a>.</p>
How to use shape information during FX pattern rewriting?

aeron: <p>I am trying to rewrite the <code>index_put</code> op with <code>scatter</code>, in this procedure, I need to use some shape information of input tensors:</p>
<pre><code class="lang-auto">def index_put_pattern(data, indices, values, accumulate):
    return torch.ops.aten.index_put(data, indices, values, accumulate)

def index_put_replacement(data, indices, values, accumulate):
    index = indices[0]
    Q, P = index.shape
    _, N = values.shape
    bidx = torch.broadcast_to(index.reshape(Q, P, 1), (Q, P, N)).reshape(Q * P, N)
    values = torch.broadcast_to(values.reshape(1, P, N), (Q, P, N)).reshape(Q * P, N)
    return data.scatter_(0, bidx, values, reduce="add")
</code></pre>
<p>But error occurs:</p>
<pre><code class="lang-auto">    bidx = torch.broadcast_to(index.reshape(Q, P, 1), (Q, P, N)).reshape(Q * P, N)
TypeError: broadcast_to(): argument 'size' (position 2) must be tuple of ints, but found element of type Proxy at pos 0
</code></pre>
<p>Any solution for this?</p>
None grads after backward()

Beyazit_Bestami_Yuks: <p>When I run this code;</p>
<h1>
<a name="build-a-feed-forward-network-1" class="anchor" href="#build-a-feed-forward-network-1"></a>Build a feed-forward network</h1>
<p>model = nn.Sequential(nn.Linear(2352, 128),<br>
nn.ReLU(),<br>
nn.Linear(128, 64),<br>
nn.ReLU(),<br>
nn.Linear(64, 10),<br>
nn.LogSoftmax(dim=1))</p>
<p>criterion = nn.CrossEntropyLoss()<br>
dataiter = iter(trainloader)<br>
images, labels = next(dataiter)<br>
images = images.view(images.shape[0], -1)</p>
<h1>
<a name="forward-pass-get-our-log-probabilities-2" class="anchor" href="#forward-pass-get-our-log-probabilities-2"></a>Forward pass, get our log-probabilities</h1>
<p>logits = model(images)<br>
logits = Variable(torch.randn(10, 120).float(),  requires_grad = True)<br>
labels = Variable(torch.FloatTensor(10).uniform_(0, 120).long())</p>
<h1>
<a name="calculate-the-loss-with-the-logits-and-the-labels-3" class="anchor" href="#calculate-the-loss-with-the-logits-and-the-labels-3"></a>Calculate the loss with the logits and the labels</h1>
<p>loss = criterion(logits, labels.squeeze())</p>
<p>than;<br>
print(‘Before backward pass: \n’, model[0].weight.grad)<br>
loss.backward()<br>
print(‘After backward pass: \n’, model[0].weight.grad)</p>
<p>I get;<br>
Before backward pass:<br>
None<br>
After backward pass:<br>
None</p>
<p>Why I’m getting None grad after backward().<br>
Could anyone help me please?</p>

ptrblck: <p>You are detaching the output of the model by rewrapping it into a new <code>Variable</code> with a new randomly initialized tensor. Remove this line and it should work. Also not that <code>Variable</code>s are deprecated since PyTorch 0.4.</p>

Beyazit_Bestami_Yuks: <p>Thanks for answer ptrblck<br>
But When I remove that line I got an error;</p>
<p>“RuntimeError: 0D or 1D target tensor expected, multi-target not supported”</p>
<p>Those are my line:</p>
<p>logps = model(images)<br>
print(images.shape)<br>
print(logps.shape)<br>
loss = criterion(logps, labels)</p>
<p>output:<br>
images.shape–&gt; torch.Size([64, 2352])<br>
logps.shape → torch.Size([64, 10])</p>

Beyazit_Bestami_Yuks: <p>I fixed in this way:</p>
<p>logps  = model(images)<br>
labels=labels.type(torch.LongTensor)<br>
loss = criterion(logps , labels.squeeze())</p>
<p>Thanks for your help…</p>
Pass model to C++ wrapped with SWIG

peastman: <p>I have a C++ package that uses SWIG to generate Python wrappers.  I want to write a function that takes a module as an argument.  In the C++ code, the argument will be a <code>torch::jit::script::Module</code>.  In the Python wrapper it will be a <code>torch.jit.ScriptModule</code>.  I found some examples of doing this with pybind11, but they seemed to rely on automatic conversion defined by PyTorch.  Since I need to use SWIG, it wasn’t clear how to adapt them.</p>
<p>If it makes the question easier, here’s an alternative way of asking it.  Suppose I’m writing C++ code.  I have a <code>PyObject</code> that represents a <code>torch.jit.ScriptModule</code>.  What do I write to get a <code>torch::jit::script::Module</code> from it?</p>
How to deal with missing labels?

mhaines94108: <p>I have created a large fashion dataset (pictures downloaded from shopping websites).  For each picture I know at least one item of clothing, but usually the person is wearing many items.  So, for example, I might have a picture of a man wearing a t-shirt, a belt, jeans, socks and shoes.  This picture has the label ‘t-shirt’.  Another picture might show a woman in a blouse and skirt and be labelled “skirt”.  The blouse is present but not labelled.</p>
<p>So I have a lot of pictures with the label “jeans” or “pants”, but there are as many, if not more, pictures that contain jeans or pants, but that have another label like “belt” or “shirt” or “tie” or “blouse” or “jacket”.  Of course, there are many pictures that don’t contain any of those items.</p>
<p>My goal is to train a model that can identify all of the clothing items present in the image.</p>
<p>Can anyone suggest strategies for dealing with this case where many labels are missing?</p>

tiramisuNcustard: <p>You might want to look at segmentation.</p>
<p><a href="https://www.kaggle.com/code/stpeteishii/torchvision-object-detection-segmentation-sample" rel="noopener nofollow ugc">https://www.kaggle.com/code/stpeteishii/torchvision-object-detection-segmentation-sample</a></p>
Passing result of torch.jit.script to C++ function

RaulPPelaez: <p>I have the following Python code:</p>
<pre><code class="lang-auto">import torch.nn as nn
import torch as pt
import myCppModule

class Model(nn.Module):
    def __init__(self):
        super().__init__()

model = pt.jit.script(Model())
myCppModule.myCppFoo(model)

</code></pre>
<p>I want to implement some function in C++, myCppFoo, that takes “model” as argument. And then expose it to python with something like pybind11<br>
I tried to implement this with:</p>
<pre><code class="lang-auto">#include&lt;torch/torch.h&gt;
#include&lt;pybind11/pybind11.h&gt;
#include&lt;iostream&gt;
void myCppFoo(const torch::jit::Module&amp; module){
  std::cerr&lt;&lt;module.get_properties().front().name&lt;&lt;std::endl;
}
PYBIND11_MODULE(myCppModule, m) {
  m.def("myCppFoo", &amp;myCppFoo);
}

</code></pre>
<p>However, when compiling the python library and running the python example, I get an error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/home/user/test/ex.py", line 10, in &lt;module&gt;
    myCppModule.myCppFoo(model)
TypeError: myCppFoo(): incompatible function arguments. The following argument types are supported:
    1. (arg0: torch::jit::Module) -&gt; None

Invoked with: RecursiveScriptModule(original_name=Model)
</code></pre>
<p>Clearly the types are not compatible. How can I modify the C++ function to accept the type provided by torch.jit.script?<br>
The docs suggest to serialize the module to a file in Python and deserialize in C++, but I would like to avoid writing an intermediate file.</p>
Converting np.complex128 datatype to torch.complex128

pankajsharma6810: <p>How we should convert an array containing np.complex128 datatype (say a complex 2d numpy array) into a torch .complex128 array of the same dimension?</p>

ptrblck: <p><code>torch.from_numpy</code> should work:</p>
<pre><code class="lang-python">a = np.random.randn(10, 10)
a = a.astype(np.complex128)
print(a.dtype)
# complex128
x = torch.from_numpy(a)
print(x.dtype)
# torch.complex128
</code></pre>

pankajsharma6810: <p>Thank you ptrblck for solving the issue. The same command was not working a few days ago and now it is working well.</p>
Id of tensors in pytorch

2h4dl: <p>Hi, everyone.<br>
My colleague found a interesting problem when check the id of some tensors.<br>
Here is the code:</p>
<pre><code class="lang-auto">import torch

x = [1, 2, 3, 4, 5, 6, 7, 8, 9]
x = torch.tensor(x, dtype=torch.float32)

pos_idxes = torch.tensor([1, 2, 3, 4], dtype=torch.int64)

idxes = pos_idxes &gt; 2

n = 10
for i in range(n):
    print(f"[{i}]", "x[pos_idxes][idxes]:", id(x[pos_idxes][idxes]), "x[pos_idxes[idxes]]:", id(x[pos_idxes[idxes]]))
</code></pre>
<p>Results:</p>
<pre><code class="lang-auto">[0] x[pos_idxes][idxes]: 2217284501312 x[pos_idxes[idxes]]: 2217284108096
[1] x[pos_idxes][idxes]: 2217284501312 x[pos_idxes[idxes]]: 2217284108096
[2] x[pos_idxes][idxes]: 2217284501312 x[pos_idxes[idxes]]: 2217284108096
[3] x[pos_idxes][idxes]: 2217284501312 x[pos_idxes[idxes]]: 2217284108096
[4] x[pos_idxes][idxes]: 2217284501312 x[pos_idxes[idxes]]: 2217284108096
[5] x[pos_idxes][idxes]: 2217284501312 x[pos_idxes[idxes]]: 2217284108096
[6] x[pos_idxes][idxes]: 2217284501312 x[pos_idxes[idxes]]: 2217284108096
[7] x[pos_idxes][idxes]: 2217284501312 x[pos_idxes[idxes]]: 2217284108096
[8] x[pos_idxes][idxes]: 2217284501312 x[pos_idxes[idxes]]: 2217284108096
[9] x[pos_idxes][idxes]: 2217284501312 x[pos_idxes[idxes]]: 2217284108096
</code></pre>
<p>Pytorch should be return a new tensor when use index, it will have a new id of new tensor, but the code’s output make me confused.</p>
<p>Why it give the same id?</p>

ptrblck: <p>Indexing a tensor will create a new Python object with an <code>id</code>, which can be reused by Python.<br>
You are not checking the <code>id</code> of any underlying tensor, its data, etc. but just from the Python object holding the reference to the indexed tensor.</p>

2h4dl: <p>Hi, ptrblck. This is what I get that I just checked the data id, not the reference object id.<br>
If a reference object  point to the tensor data, <code>id</code> will give the address of reference object, right?</p>
Using vision/references/detection/train.py with augmentation training mAP is abnormal

Middle_Hsu: <p>The offical vision/references/detection/train.py using COCO api to calculate mAP.<br>
When it not using data augmentation that train mAP is normal,<br>
Training one epoch like that:<br>
train mAP is 0.25383283202173673<br>
validation mAP is 0.30275321883612916</p>
<p>However, I turned RandomHorizontalFlip on and probability setting as 0.5.<br>
Training one epoch like that:<br>
train mAP is 0.03392322767009024<br>
validation mAP is 0.18803215716046826</p>
<p>Futhermore, I turned RandomHorizontalFlip on and probability setting as 1.0.<br>
Training one epoch like that:<br>
train mAP is 0.24607017764763847<br>
validation mAP is 0.24640787952490323</p>
<p>I thought RandomHorizontalFlip with Annotation transfer is nothing wrong. What others problem do I meet?</p>
Do I need to pre-train Discriminator in eval()?

Peterwisu: <p>Hi, I am trying to train a GAN-based model with a pre-train discriminator. So, I wonder do I need to put my pre-train discriminator in the evaluation stage or if I do not have to. because the parameter optimizer optimisation is only from generator and I look through example training GAN in PyTorch <a href="https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">DCGAN Tutorial — PyTorch Tutorials 1.13.1+cu117 documentation</a>. In this example, they didn’t put discriminator in eval() when updating the generator so is it the same as pre-train?? Thank you</p>

J_Johnson: <p>You want the gradients to come from the Discriminator. That’s why for training the Generator(the second discriminator pass), you’ll note they did not <code>.detach()</code> on the fake images. In this way, you can think of the updated Discriminator and Generator behaving as one model and by setting the labels to real, we get the difference between the fake images and what the Discriminator identified as “real” features. So then the Discriminator communicates to the Generator what parts of the images looked fake.</p>

Peterwisu: <p>SO that means to get the gradient, I should not put the train discriminator in eval, right? Thank you</p>

J_Johnson: <p>Sorry, I may have misunderstood your question. <code>eval()</code> will disable any batchnorm or dropout layers and has nothing to do with gradients. The correct way to run inference without gradients is:</p>
<pre><code class="lang-auto">with torch.no_grad():
    output=model(input)
</code></pre>
<p>With that said, you could use <code>.eval()</code> and that might possibly help speedup the Generator’s training, as it will turn off the batchnorm layers on the Discriminator. But do not use <code>torch.no_grad()</code>.</p>
How can I figure out the GPU memory overhead of an optimizer?

Vedant_Roy: <p>I’m interested in figuring out how much memory, an optimizer like Adam, adds compared to a no-overhead optimizer (like plain SGD).</p>
<p>Current plan is the following:</p>
<pre><code class="lang-auto">    print("START:")
    print(torch.cuda.memory_summary())

    with torch.cuda.amp.autocast(enabled=True):
        for _ in range(5):
            print("BEFORE BATCH:")
            print(torch.cuda.memory_summary())
            batch = torch.randn(2, 3, 224, 224).cuda()

            loss = model(batch)
            loss.backward()
            optimizer.zero_grad()
            loss_val.backward()
            optimizer.step()

            print("AFTER BATCH:")
            print(torch.cuda.memory_summary())
    print(torch.cuda.memory_summary())
</code></pre>
<p>However, I don’t like this plan because it’s not very precise. I think there should be a better way to measure the GPU memory overhead an optimizer is adding.</p>
Why is target sequence masked with triangular matrix?

czylox: <p>Hi,</p>
<p>In typical sequence to sequence task and language modeling task, the decoder is usually masked with triangular matrix to enforce that any word on the target sequence can only attend to the words before it. Is it mainly for computational efficiency?</p>
<p>E.g.: My target sequence is ABCDEFG (source doesn’t matter). When the decoder tries to predict F, why can’t A/B/C/D/E attend to each other to create a better context for F?<br>
One obvious drawback of this approach is that to predict the next word G, all previous attention needs to be recomputed. So is this why we mask target sequence in the decoder to only allow single direction attention? If we allow the already decoded target tokens to attend to each other, we could achieve better prediction for the next token (with much higher cost though), right?</p>
<p>Thanks</p>