akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
[TorchDynamo] Too many function args in a fx graph

vivek_khandelwal: <p>Hi all, I’m working on compiling a model using Torchdynamo + <a href="https://github.com/llvm/torch-mlir" rel="noopener nofollow ugc">Torch-MLIR</a>, and running it via <a href="https://github.com/iree-org/iree/" rel="noopener nofollow ugc">IREE</a>. The issue I’m facing is that the fx graph that is getting generated contains nearly 1000 function args (all tensors). Because of such a large number of function args, it results in a segmentation fault during IREE runtime. The reason is that the system heap gets exhausted when allocating such a large number of tensors (which are large in size). FYI, these tensors are the weights of the model. I have two questions here:<br>
1.) Can we make any changes to the fx graph such that these function args are replaced by a single arg which would be a list of tensors and all the usages of these function args with the list accesses?<br>
2.) Is there any other way we can handle this type of instance via torchdynamo?</p>

marksaroufim: <p>cc <a class="mention" href="/u/sherlocknomad">@SherlockNoMad</a> who might know</p>

SherlockNoMad: <p>You are getting a FX graph produced by AOTAutograd, which lifts parameters/buffers as graph inputs. You may ask why we wanna do this? This is because we want to produce a stateless graph, which will be easier to work with for most backends.</p>
<p>As a work around, you may want to write a pass that fold inputs back into graph as its internal state.</p>
Pytorch 2.0 compile warning meaning

jay_thakur: <p>I see that there is warning when run a model using torch.compile(model) way</p>
<p>Warning:<br>
2128922-2023-01-18 10:11:40:751  INFO     [logging.py:121] — Step 1: torchdynamo start tracing forward<br>
2128922-2023-01-18 10:11:40:756  INFO     [logging.py:121] — Step 1: torchdynamo done tracing forward (RETURN_VALUE)<br>
2128922-2023-01-18 10:11:40:757  INFO     [logging.py:121] — Step 2: calling compiler function debug_wrapper<br>
2128922-2023-01-18 10:11:40:884  INFO     [logging.py:121] — Step 3: torchinductor compiling FORWARDS graph 0<br>
2128922-2023-01-18 10:11:40:887  WARNING  [graph.py:280] — Creating implicit fallback for:<br>
target: aten._adaptive_avg_pool3d.default<br>
args[0]: TensorBox(StorageBox(<br>
InputBuffer(name=‘primals_1’, layout=FixedLayout(‘cpu’, torch.float32, size=[8, 2048, 7, 7], stride=[100352, 49, 7, 1]))<br>
))<br>
args[1]: [32, 7, 7]<br>
2128922-2023-01-18 10:11:40:894  WARNING  [ir.py:2942] — Using FallbackKernel: torch.ops.aten._adaptive_avg_pool3d.default</p>
<p>can someone help me to understand above the reason of fallback? is it running in eager mode?<br>
what can be done to execute in compile mode?</p>

marksaroufim: <p>Yes fallback means it’s falling back to eager mode, you can add a decomposition for it so that’s no longer the case - it’s a good first issue <a href="https://github.com/pytorch/pytorch/issues/93757" class="inline-onebox" rel="noopener nofollow ugc">TorchInductor missing ops tracker · Issue #93757 · pytorch/pytorch · GitHub</a></p>
No speedup from torch.complile()

mileseverett: <p>Hey, just wondering why I’m getting almost no speedup from torch.compile in this toy example</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://colab.research.google.com/drive/1DdtTo6FSoEnaEn7Qr_1_KuO68iLIzuUP?usp=sharing">
  <header class="source">
      <img src="https://ssl.gstatic.com/colaboratory-static/common/772f6810a580d9ca9164d193a55931b3/img/favicon.ico" class="site-icon" width="128" height="128">

      <a href="https://colab.research.google.com/drive/1DdtTo6FSoEnaEn7Qr_1_KuO68iLIzuUP?usp=sharing" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>

  <article class="onebox-body">
    <img src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar" width="260" height="260">

<h3><a href="https://colab.research.google.com/drive/1DdtTo6FSoEnaEn7Qr_1_KuO68iLIzuUP?usp=sharing" target="_blank" rel="noopener nofollow ugc">Google Colaboratory</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>With T4 GPU assigned.</p>
<p>Am I doing something wrong? I know it is a small example but should there not be at least some speedup?</p>
<p>Thanks<br>
Miles</p>

ptrblck: <p>You might want to read through <a href="https://pytorch.org/docs/master/compile/faq.html#why-am-i-not-seeing-speedups">this section</a> of the docs. CC <a class="mention" href="/u/marksaroufim">@marksaroufim</a></p>

marksaroufim: <p>Unfortunately T4 chips won’t benefit as much from torch.compile because torch.compile primarily helps with memory bandwidth bound workloads as in workloads where the GPU is so fast that you can’t transfer data quickly into it, this is less of a problem on older GPUs since they’re slower. In addition newer GPUs also introduced tensor cores which is so critical for performance nowadays</p>
<p>So my suggestion is to find a notebook provider where you can pick your GPU and either get an A10G or A100</p>
I'm trying to understand LayerNorm

John_Deatherage: <p>I’m trying to wrap my head around how to use nn.LayerNorm().  As I understand it, Layer Normalization takes the weights of a hidden layer and rescales them around the mean and standard deviation. Correct so far?</p>
<p>For example, let’s assume a simple plain vanilla feed-forward network.<br>
def <strong>init</strong>(self, input_size, neurons, num_classes):<br>
super(NeuralNet, self).<strong>init</strong>()<br>
self.fc1 = nn.Linear(25, 10, bias=True) # input layer with 25 features<br>
self.fc2 = nn.Linear(10, 10, bias=True) # hidden layer with 10 neurons<br>
self.fc3 = nn.Linear(10, 2, bias=True)   # output layer with 2 outputs</p>
<pre><code>    def forward(self, x):
    x = F.relu(self.fc1(x))
    x = F.relu(self.fc2(x))   
    return torch.sigmoid(self.fc3(x))
</code></pre>
<p>I want to add nn.LayerNorm to the layers  (fc1 &amp; fc2).  Should I add it into the forward method?</p>
<pre><code>    def forward(self, x):
    x = F.relu(self.fc1(x))
    x = self.fc1.nn.LayerNorm(?????)
    x = F.relu(self.fc2(x))  
    x = self.fc2.nn.LayerNorm(?????) 
    return torch.sigmoid(self.fc3(x))
</code></pre>
<p>What I don’t understand are the arguments/parameters needed… I’ve read the documentation: torch.nn.LayerNorm(<em>normalized_shape</em>, <em>eps=1e-05</em>, <em>elementwise_affine=True</em>, <em>device=None</em>, <em>dtype=None</em> )  Using my example, what is the normalized_shape for fc1?  for fc2?</p>
<p>be gentle, I’m a newbie  THANK YOU for taking the time to read this and to HELP ME!</p>
Do gradients propagate through forward hooks?

Pericules: <p>For a binary classification problem using a feedforward NN with 5 layers, I want to create a joint loss function that includes predictive outputs from the intermediate layers. I am considering what is the best way to access the outputs of these intermediate layers, and one approach would be to add hooks to them. If I were to add a hook to an intermediate layer, and pass the output of that layer through a linear layer and to a loss function, would the loss.backward() propagate through these hooks properly?</p>

ptrblck: <p>Yes, this should be the case as seen in this small example which uses a hook to get an intermediate output, uses another custom layer, calculates the loss, and backpropagates through all used layers:</p>
<pre><code class="lang-python">act = {}
def get_hook(name):
    def hook(m, input, output):
        act[name] = output
    return hook

model = models.resnet18()
model.layer1[1].conv2.register_forward_hook(get_hook("layer1_conv2"))

x = torch.randn(1, 3, 224, 224)
out = model(x)

# verify that no grads are set
for name, param in model.named_parameters():
    print("param {}, grad {}".format(name, param.grad))
# param conv1.weight, grad None
# param bn1.weight, grad None
# param bn1.bias, grad None
# ...
# param layer4.1.bn2.bias, grad None
# param fc.weight, grad None
# param fc.bias, grad None


custom_layer = nn.Conv2d(64, 1, 3, 1, 1)
act_out = custom_layer(act["layer1_conv2"])

loss = act_out.mean()
loss.backward()

# the custom layer as well as all layers used to create layer1_conv2 output should have valid grads
for name, param in custom_layer.named_parameters():
    grad = param.grad.abs().sum() if param.grad is not None else None
    print("param {}, grad {}".format(name, grad))
# param weight, grad 222.42572021484375
# param bias, grad 1.000019907951355

for name, param in model.named_parameters():
    grad = param.grad.abs().sum() if param.grad is not None else None
    print("param {}, grad {}".format(name, grad))
# param conv1.weight, grad 30.616352081298828
# param bn1.weight, grad 0.10833178460597992
# param bn1.bias, grad 0.06121913343667984
# param layer1.0.conv1.weight, grad 23.733787536621094
# param layer1.0.bn1.weight, grad 0.046802934259176254
# param layer1.0.bn1.bias, grad 0.03454311192035675
# param layer1.0.conv2.weight, grad 22.624469757080078
# param layer1.0.bn2.weight, grad 0.036605916917324066
# param layer1.0.bn2.bias, grad 0.04052506014704704
# param layer1.1.conv1.weight, grad 37.31376266479492
# param layer1.1.bn1.weight, grad 2.3493189811706543
# param layer1.1.bn1.bias, grad 2.9538869857788086
# param layer1.1.conv2.weight, grad 902.742919921875
# param layer1.1.bn2.weight, grad None
# param layer1.1.bn2.bias, grad None
# param layer2.0.conv1.weight, grad None
# ...
# param layer4.1.bn2.bias, grad None
# param fc.weight, grad None
# param fc.bias, grad None
</code></pre>

Pericules: <p>Excellent, thanks a lot for the example! This is precisely what I was after.</p>
Batched elementwise multiplication question

Nyakov: <p>I have <em>input_tensor</em> of shape <code>[batch, num_samples, num_features]</code></p>
<p>And <em>attention_tensor</em> of shape <code>[batch, num_samples]</code></p>
<p>Now I need, for every <em>batch</em>, for every <em>sample</em>, multiply elementwise value from <em>attention_tensor</em> to each <em>feature</em> value in <em>input_tensor</em> so output tensor will have shape <code>[batch, num_samples, num_features]</code></p>
<p>How can I do this in Pytorch with minimum overhead?</p>

soulitzer: <p>Does <code>input_tensor * attention_tensor.unsqueeze(-1)</code> work?</p>

Nyakov: <p>Yes it works!<br>
As well as <code>input_tensor * attention_tensor.unsqueeze(2)</code> - what I figured out by experiment.</p>
<p>Although I am not sure if this solution is optimal.</p>
Clear all tesnors saved for backward (reboot)

ralzq01: <p>Hi I try to release the cuda memory when I try to handle the following exception due to OOM error:</p>
<pre><code class="lang-python">try:
    output = model(input)
    output.backward()
except Exception as e:
    # I want to clear all tensors saved for backward here
    pass
</code></pre>
<p>When memory is not sufficient for training (OOM exception), some already forwarded tensor activations are kept in autograd engine (and thus consume lots of memory). I want to clear them and restart with another model. I tried using <code>torch.cuda.empty_cache()</code>, but it doesn’t take effect. Is there any way to release these tensors?</p>

soulitzer: <p><code>del output</code> should clear the graph since the output is the only thing keeping the graph alive, and the saved variables stored on the graph should be released as well as a result</p>
TransformerEncoder truncates output when some token positions are masked by `src_key_padding_mask` across batch

jasperhyp: <p>When <code>src_key_padding_mask</code> ([N, K]) has one column (k) all with value <code>True</code>, <code>TransformerEncoder</code> will remove this column in the output, causing inconsistency issues.  Minimal example:</p>
<p>Class init</p>
<pre><code class="lang-python">encoder_layer = nn.TransformerEncoderLayer(d_model=256, nhead=4, dim_feedforward=512, activation='gelu', norm_first=False, batch_first=False)
self.transformer_encoder = nn.TransformerEncoder(encoder_layer, num_layers=3, enable_nested_tensor=True)
</code></pre>
<p>Forward</p>
<pre><code class="lang-python">x = torch.randn(2000, 6, 256)
mask = torch.ones(2000, 6)
mask[:, 0] = 0  # here I masked 5 columns instead of just one
mask = mask.bool()
out = self.transformer_encoder(src = x, src_key_padding_mask = mask)
assert out.shape[1] == 6  # Error, the actual dim is only 1
</code></pre>
<p>I don’t think this is a good practice because oftentimes we want aggregation on the sequence level, and this kind of removal would make the use of scatter aggregation functions, e.g. <code>scatter_mean</code> difficult.</p>

jasperhyp: <p>Update: This relates to the option <code>enable_nested_tensor</code>.  When setting it to <code>False</code> things become fine.  I met tons of problems when using nester tensor and fast paths (including a previous <a href="https://discuss.pytorch.org/t/register-forward-hook-doesnt-work-for-nestedtensor/158374">post</a>) – I think it would be very very helpful if PyTorch can instead set this to False by default.  It’s just not as stable as it should be.</p>

ptrblck: <p>Could you create an issue on <a href="https://github.com/pytorch/pytorch/issues">GitHub</a> describing how the usage of <code>NestedTensor</code> causes issues in your use case, please?</p>

jasperhyp: <p>Sure thanks, created one already</p>
Error:torch.cuda.is_available() return true in cmd but return false in pycharm

qianlifanyue: <p>In cmd, I printed “import torch” then “torch.cuda.is_isavailable()”, it returned true;<br>
but at python console in pycharm I got false.<br>
I have set the right python interpreter, and there is ‘pytorch’ under the package menu. Are there any possible causes? Thanks in advance.</p>

ptrblck: <aside class="quote no-group" data-username="qianlifanyue" data-post="1" data-topic="175468">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/qianlifanyue/48/58618_2.png" class="avatar"> qianlifanyue:</div>
<blockquote>
<p>I have set the right python interpreter, and there is ‘pytorch’ under the package menu. Are there any possible causes?</p>
</blockquote>
</aside>
<p>You are most likely using different Python environments and the one used in PyTorch might have installed the CPU-only binary.<br>
Also, according to <a href="https://discuss.pytorch.org/t/torch-cuda-is-available-returns-false-debugging-but-true-when-execute-on-the-console/173745/8">this topic</a> the user claimed <code>os.environ["CUDA_VISIBLE_DEVICES"]='1'</code> was set somehow in Pycharm without manually setting it, which was hiding the only available GPU.</p>

qianlifanyue: <p>I see the python envirment I set  is …\anaconda3\python.exe, you mean this is python2 but not python3? I see the environment in cmd is python 3.10 indeed.But what is CPU-only binary?</p>

ptrblck: <p>No, I meant different virtual Python environments which allow you to isolate installed packages.<br>
“CPU-only” binary refers to the pip wheel or conda binary shipping with CPU kernels only without any CUDA kernels or CUDA libraries.</p>

qianlifanyue: <p>Thanks a lot. I find where is wrong. My pytorch Build is py3.9_cpu_0. I will uninstall it and try again.</p>
Checkpoints loading error into ResNet50 model

Muhammad_Faizan: <p>Hi, I am loading trained checkpoints on person reid dataset like Market1501. I am using <a href="https://github.com/mikwieczorek/centroids-reid" rel="noopener nofollow ugc">this</a> repository for reid which is top on the leader board in reid. I am running inference on image folder when I run the inference command from <a href="https://github.com/mikwieczorek/centroids-reid/tree/main/inference" rel="noopener nofollow ugc">here</a>, it gives me the following error.</p>
<pre><code class="lang-auto">File "/home/facit/Faizan/reid/ctl/centroids-reid/modelling/backbones/resnet.py", line 157, in load_param
    self.state_dict()[new_param_name].copy_(param_dict[i])
KeyError: 'center_loss.centers'

</code></pre>
<p>I think there is no key named center_loss.centers in state_dict, I am not sure how to solve this issue. How you please give some insight on it. Complete checkpoints loading code is given below for more information:</p>
<pre><code class="lang-auto">def load_param(self, model_path):
        param_dict = torch.load(model_path)
        if 'state_dict' in param_dict:
            param_dict = param_dict['state_dict']
        for i in param_dict:
            if 'backbone' in i:
                new_param_name = i[14:]
            elif 'base' in i:
                new_param_name = i[5:]
            else:
                new_param_name = i
            if 'fc' in i:
                continue
            if 'bottleneck' in i:
                continue
            if 'classifier' in i:
                continue
            if 'transformer' in i:
                continue
            # print(type(self.state_dict()))
            # print('\n')
            # print(f'keys of state dict: {(self.state_dict().keys())}')
            self.state_dict()[new_param_name].copy_(param_dict[i])
</code></pre>
<p>here is the <a href="https://github.com/mikwieczorek/centroids-reid/blob/main/modelling/backbones/resnet.py" rel="noopener nofollow ugc">resnet model file</a>.</p>
Freeze weights or constraint the optimizer?

sanparith_marukatat: <p>Hello,</p>
<p>Consider the following 2-layer NN:</p>
<pre><code class="lang-auto">model = nn.Sequential(OrderedDict([
          ('1', nn.Linear(3,3)),
          ('2', nn.Linear(3,3)),
        ]))
</code></pre>
<p>Suppose that I want to freeze the second layer, and train only the first layer.<br>
I think there are two methods to achieve this</p>
<ol>
<li>Set ‘require_grad’ of the second layer to False then train</li>
</ol>
<pre><code class="lang-auto">for param in model[1].parameters():
    param.requires_grad = False
optimizer = optim.AdamW(model.parameters(), lr=0.0001)
</code></pre>
<ol start="2">
<li>Constraint the optimizer to work with parameters of the first layer:</li>
</ol>
<p><code>optimizer = optim.AdamW(model[0].parameters(), lr=0.0001)</code></p>
<p>Both methods yield the same value for the parameters of the first layer.<br>
So, which method is preferred? or should we combine both:</p>
<pre><code class="lang-auto">for param in model[1].parameters():
    param.requires_grad = False
optimizer = optim.AdamW(model[0].parameters(), lr=0.0001)
</code></pre>

Joschka: <p>One thing that would speak in favor of explicitly setting <code>requires_grad = False</code> for frozen parameters is the decrease in overhead. By doing so you effectively tell torch to not accumulate gradients for those leaf nodes during the backward pass, whereas with option 2 they are computed but unused.</p>
<p>The <a href="https://pytorch.org/docs/stable/notes/autograd.html" rel="noopener nofollow ugc">autograd documentation</a> is also straightforward on this subject:</p>
<blockquote>
<p>Setting <code>requires_grad</code> should be the main way you control which parts of the model are part of the gradient computation, for example, if you need to freeze parts of your pretrained model during model fine-tuning.</p>
</blockquote>
Weight in CNN U-Net with pretrained encoder

Anastasia_Berlianna: <p>hi, i’m build unet architecture with pretrained resnet34 as encoder, after i train the model, i printed each of the layer’s weight and the result is</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/f/af8287ba55c5229d01ab3c7fb934714d153c5f4d.png" alt="image" data-base62-sha1="p2Dds0HB9tTV4NzkIeTD5dh4qAt" width="462" height="179"></p>
<p>i’m still clueless how is the weight affecting the unet performance? is negative weight consider bad? i’m still newbie in deep learning, thankyou!</p>
Does state-of-the-art image classification models increase object detection accuracy

Mahi: <p>Implementing object detection using state-of-the-art image classification models increase performance?. The implementation of NMS are taken care of.<br>
Implementation structure<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/f/4/f4443deef6d34add66b1c5c1f0016a615d6a4339.png" alt="image" data-base62-sha1="yQSI2WWLULfBbNLZtdT8s1E81FD" width="506" height="452"></p>
Error when jit.trace with dictionary input

ecolss: <p>Is it possible to use dict as input to traced model, and the dict key-value has different value types?<br>
For example, the following dummy example failed with an error</p>
<pre><code class="lang-auto">class A(th.nn.Module):
    def __init__(self):
         super().__init__()
         self.lin = th.nn.Linear(3,2)

    def forward(self, data):  # data is a dict
         return self.lin(data["x"])


data = {
    "x": th.rand(2,3), 
    "y": {
        "z": th.rand(5,3)    # "y" has a dict as value, but "x" has a tensor as value
    }
}

a = A()
a(data)   # works well

th.jit.trace(a, data)  # error happens
</code></pre>
<p>Error says</p>
<pre><code class="lang-bash">RuntimeError: Tracer cannot infer type of ({'x': tensor([[0.1657, 0.9904, 0.5861],
        [0.2403, 0.2352, 0.6989]]), 'y': {'z': tensor([[0.1907, 0.8666, 0.1840],
        [0.6437, 0.2888, 0.0081],
        [0.6681, 0.7088, 0.3869],
        [0.4880, 0.8989, 0.3832],
        [0.0746, 0.7623, 0.2115]])}},)
:Dictionary inputs to traced functions must have consistent type. Found Tensor and Dict[str, Tensor]
</code></pre>

ptrblck: <p>Based on <a href="https://github.com/pytorch/pytorch/issues/75012#issuecomment-1304263445">this issue and workaround</a> it seems nested <code>dict</code>s are not allowed and you might need to <code>flatten/unflatten</code> the input.</p>
The affects of input size, semantic segmentation

patatessever: <p>Hi, I asked a few questions before about a model I am working on. I am wondering how the input size affects the overall performance. For example, my image’s size is 1024x1024. If I create 4 512x512 tiles from these images for one training and create 1024 32x32 tiles for another training should I expect any changes in results? Does batch size also have an effect on this? My theory is that it can only affect the number of epochs before converting but I want to be sure about this.  And one last question, can I use a model trained by 32x32 images for a prediction, in which the input is 256x256 for example (I can but is this affects the results ? ).</p>

ptrblck: <aside class="quote no-group" data-username="patatessever" data-post="1" data-topic="175383">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/patatessever/48/57624_2.png" class="avatar"> patatessever:</div>
<blockquote>
<p>For example, my image’s size is 1024x1024. If I create 4 512x512 tiles from these images for one training and create 1024 32x32 tiles for another training should I expect any changes in results?</p>
</blockquote>
</aside>
<p>I would claim it might depend on the input resolution as e.g. the conv kernels were trained to detect features from a specific resolution while they might fail to detect the same features if you “zoom in/out”, which is how resizing could also be seen. If the input resolution is the same and only tiles were created, I would expect to see differences at their borders due to padding or a smaller output size assuming you are working on e.g a segmentation use case. For a classification use case I can imagine the model might completely fail to detect valid objects from a single <code>32x32</code> tile assuming the object spans the full input of <code>1024x1024</code>.</p>
<aside class="quote no-group" data-username="patatessever" data-post="1" data-topic="175383">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/patatessever/48/57624_2.png" class="avatar"> patatessever:</div>
<blockquote>
<p>Does batch size also have an effect on this?</p>
</blockquote>
</aside>
<p>The batch size has generally affects the overall model training and would influence a few layers, such as batchnorm layers, which update their running stats from the input activations.</p>
<aside class="quote no-group" data-username="patatessever" data-post="1" data-topic="175383">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/patatessever/48/57624_2.png" class="avatar"> patatessever:</div>
<blockquote>
<p>And one last question, can I use a model trained by 32x32 images for a prediction, in which the input is 256x256 for example (I can but is this affects the results ? ).</p>
</blockquote>
</aside>
<p>Same as before: I claim it depends on the use case.</p>
Trying to use triton on torch inductor

aldesilv: <p>Trying to use the triton kernel for this convolution, however I get an error. Am I doing something wrong?</p>
<pre><code class="lang-auto">import torch

import torch._dynamo
from torch._inductor import config

@config.patch({"triton.convolution" : "triton"})
def run_conv():
    class M(torch.nn.Module):
        def __init__(
            self,
            **kwargs,
        ):
            super().__init__()
            self.conv1 = torch.nn.Conv2d(
                             4,
                             320,
                             (3,3),
                             (1,1),
                             (0,0),
                             (1,1),
                         )

        def forward(self, x):
            x1 = self.conv1(x)
            return x1

    mod = M().cuda()

    opt_mod = torch._dynamo.optimize("inductor")(mod)
    inputs = (
        torch.randn([2, 4, 66, 66], dtype=torch.float32, device="cuda"),
    )
    opt_y = opt_mod(*inputs)
    print(opt_y)

run_conv()
</code></pre>
<p>…</p>
<pre><code class="lang-auto">File "/home/alexander/triton/python/triton/runtime/autotuner.py", line 95, in prune_configs
    pruned_configs = self.early_config_prune(self.configs, self.nargs)
  File "/home/alexander/pytorch/torch/_inductor/triton_ops/conv_perf_model.py", line 100, in early_config_prune
    cc = _triton.runtime.cc(backend, device)
AttributeError: module 'triton._C.libtriton.triton.runtime' has no attribute 'cc'
</code></pre>
<p>in “torch/_inductor/triton_ops/conv.py”,<br>
I tried to remove the 'return’s in the <span class="mention">@triton.jit</span> functions<br>
and to not use the autotuner, and I had to remove the ‘tl.store’ in _kernel_delta_x() or it would get stuck</p>
<pre><code class="lang-auto">tl.store(y_ptrs, acc, mask=mask_y)

 return
</code></pre>
<p>but I’m not sure what is wrong with the tl.store.</p>

GLZhu: <p>Hi, there is a related <a href="https://github.com/pytorch/pytorch/issues/95775" rel="noopener nofollow ugc">issue</a> here. From the link it seems that <span class="mention">@jansel</span> is working on this.<br>
This is a version mismatch between triton and pytorch and believe it will be resolved in later version.</p>
List to group results in forward method

mgiordy: <p>Hi all!</p>
<p>I have a quick “validation” question. Given n parallel model heads, which do not share parameters and should be trained on different data, is it ok to append the outputs in a list and then call torch.cat on the list?<br>
Here’s few lines of code to better illustrate the problem:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

class DenseNet(nn.Module):
    def __init__(self):
        super(DenseNet, self).__init__()
        self.dense_list = nn.ModuleList([nn.Linear(16, 16) for i in range(3)])

    def forward(self, x):
        tcn_out = [] # torch.empty(0, self.num_channel_out*self.seq_lenght)
        for i, dense_i in enumerate(self.dense_list):
            o = dense_i(x[:,i,:])
            tcn_out.append(o)
        return torch.cat(tcn_out, dim=1)

input = torch.rand((32, 3, 16))

dense_net = DenseNet()
out = dense_net(input)
print(out.shape)
</code></pre>
<p>My concern is possible confusions during the back propagation due to the mixup of tensor and list.</p>
<p>Thanks for your time!</p>

ptrblck: <p><code>torch.cat</code> should work and will not confuse the tensors during the backward pass as seen here:</p>
<pre><code class="lang-python">modules = [nn.Linear(1, 1, bias=False) for _ in range(3)]
modules[0].weight.requires_grad = False
modules[2].weight.requires_grad = False

x = torch.randn(1, 3, 1)

out = []
for idx in range(len(modules)):
    x_ = x[:, idx]
    o = modules[idx](x_)
    out.append(o)
out = torch.cat(out, dim=1)
print(out.shape)
# torch.Size([1, 3])
out.mean().backward()

for module in modules:
    print(module.weight.grad)
# None
# tensor([[-0.0284]])
# None
</code></pre>
Implementing a new dataset to CV architecture

zehranrgi: <p>Hi,</p>
<p>I want to implement the <a href="https://github.com/aras62/PIE" rel="noopener nofollow ugc">PIE dataset</a> in the <a href="https://github.com/Khrylx/AgentFormer" rel="noopener nofollow ugc">AgentFormer arch</a>.<br>
AgentFormer uses ETH and nuScene datasets. I successfully run these datasets on this arch. However, I couldn’t take a good way with the PIE dataset. I am not sure how I could write a new data loader for it. Which steps should I take?<br>
I normally have experience in implementing articles without looking at similar GitHub codes, but this time I stuck.</p>
<p>Thank you for any help.</p>
Control GPU Memory cache

Iamidiot: <p>I am using U-net modified as 3D Convolution version. The problem here is that pytorch takes a very large memory size for my 3D U-net. Initially, I only set the training batch size to 2 because of this problem.</p>
<p><code>torch.cuda.memory_allocated()</code> outputs low memory usage (around 5GB), but <code>torch.cuda.max_memory_allocated()</code> outputs high memory usage (around 36GB).</p>
<p>At first, I thought it was a problem caused by the excessively large size of 3D features stored due to U-net’s skip connection, but it wasn’t. I found that just by performing a 3D convolution of size 128 x 128 x 128, Pytorch allocates a huge cache in memory that is not used by subsequent processes.</p>
<p>In order to build the network deeper and increase the number of batches, I solved the problem by putting <code>torch.cuda.reset_max_memory_allocated(device)</code>, <code>torch.cuda.empty_cache()</code>  in the middle of the forward() function, but I don’t think this is nice solution…<br>
Doing <code>torch.cuda.empty_cache()</code> slows down the network.</p>
<ol>
<li>Why does this cache problem occur?</li>
<li>Is there any better solution?</li>
</ol>

soulitzer: <p>Maybe what you are seeing is how PyTorch manages memory using the CUDA caching allocator. Even if the program does not subsequently use that region of memory, the memory already allocated by PyTorch would not be returned to the device to avoid excessive cudamalloc/free calls every iteration.</p>
<p>Calling empty_cache can be used to alleviate memory fragmentation, which may also be an issue sometimes, but it does slow down your iterations yes.</p>
<p>Autograd needs to save tensors for backward in order to recompute the backward pass (but not sure what you mean by “not used by subsequent processes”), so if you want to scale your model but are bottlenecked by memory usage, another solution is to use activation checkpointing, which allows you to save fewer activations for backward during forward in exchange for having to recompute them during backward.</p>
<p>Further reading:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://zdevito.github.io/2022/08/04/cuda-caching-allocator.html">
  <header class="source">

      <a href="https://zdevito.github.io/2022/08/04/cuda-caching-allocator.html" target="_blank" rel="noopener nofollow ugc" title="04:55AM - 04 August 2022">Zach’s Blog – 4 Aug 22</a>
  </header>

  <article class="onebox-body">
    

<h3><a href="https://zdevito.github.io/2022/08/04/cuda-caching-allocator.html" target="_blank" rel="noopener nofollow ugc">A guide to PyTorch’s CUDA Caching Allocator</a></h3>

  <p>A guide to PyTorch’s CUDA Caching Allocator</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p><a href="https://pytorch.org/docs/stable/checkpoint.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/checkpoint.html</a></p>
How can I tell whether the output of one subgraph is the input to the next?

AleksKnezevic: <p>Hi everyone,</p>
<p>I’m trying to write a custom backend for pytorch 2.0 compile and I have a question. Say I have the following code:</p>
<pre><code class="lang-auto">    class DoubleLinear(nn.Module):
        def __init__(self):
            super().__init__()
            self.l1 = nn.Linear(32, 32, bias=True)
            self.l2 = nn.Linear(32, 32, bias=False)

        def forward(self, x1, x2):
            m1 = self.l1(x1)
            print (m1)
            m2 = self.l2(x2)

            return m1 + m2, m1
</code></pre>
<p>This gets divided into two subgraphs:</p>
<pre><code class="lang-auto">pybuda/test/test_pt_2_0.py::test_torch opcode         name     target                   args            kwargs                         
-------------  -------  -----------------------  --------------  --------                                                              
placeholder    x1       x1                       ()              {}                                                                    
call_module    self_l1  self_l1                  (x1,)           {}                                                                    
call_function  add      &lt;built-in function add&gt;  (self_l1, 1.0)  {}                                                                    
output         output   output                   ((add,),)       {}       
</code></pre>
<p>and</p>
<pre><code class="lang-auto">opcode         name     target                   args           kwargs                                                                 
-------------  -------  -----------------------  -------------  --------                                                               
placeholder    x2       x2                       ()             {}                                                                     
placeholder    m1       m1                       ()             {}                                                                     
call_module    self_l2  self_l2                  (x2,)          {}                                                                                                                                                                                                            
call_function  add      &lt;built-in function add&gt;  (m1, self_l2)  {} 
output         output   output                   ((add,),)      {}   
</code></pre>
<p>How can I tell that input m1 in the second subgraph is the output of the first?</p>
Classification in C++

Sylvain_Ard: <p>Hello,<br>
do you know a classification code in C++ please ?<br>
Thank you<br>
Best regards</p>

bezdarnost: <p>Hi,<br>
You can use torch in c++<br>
Just use <code>#include &lt;torch/torch.h&gt;</code></p>

Sylvain_Ard: <p>and have you a documentation on pytorch in C++ please ?</p>

Sylvain_Ard: <p>I found it here <a href="https://pytorch.org/cppdocs/" class="inline-onebox" rel="noopener nofollow ugc">PyTorch C++ API — PyTorch master documentation</a> thank you</p>
Assign elements to arbitrary positions of tensor from another indexed tensor

manurare: <p>Currently the bottleneck of my model comes from these lines:</p>
<pre><code class="lang-auto">outputs = inputs.new_zeros(inputs.size(0), self.in_channels, self.padded_height_in, self.padded_width_in)

outputs[:, :, self.ind_ty, self.ind_tx] = inputs[:, :, self.ind_sy, self.ind_sx]
outputs[:, :, self.ind_up_ty, self.ind_up_tx] = inputs[:, self.ind_up_sc, self.ind_up_sy, self.ind_up_sx]
outputs[:, :, self.ind_down_ty, self.ind_down_tx] = inputs[:, self.ind_down_sc, self.ind_down_sy, self.ind_down_sx]
</code></pre>
<p>Is there any faster way of assigning indexed elements from <code>inputs</code> to arbitrary indexed elements of <code>outputs</code>?</p>
<p>I know gather is <a href="https://github.com/HuguesTHOMAS/KPConv-PyTorch/issues/132" rel="noopener nofollow ugc">sometimes faster</a> (specially in the backward pass) than advanced indexing, but I cannot use gather in this case since I am assigning to arbitrary position of <code>outputs</code> instead of to the full <code>outputs</code>.</p>
<p>The pytorch operators taking longer are <code>index</code> and <code>index_put_</code></p>
2D tensor multiplication with its transpose - shouldn't it return a symmetric 2D tesnor?

alywonder: <p>I am multiplying a large 2D tensor with its transpose. I am expecting that I would get a symmetric matrix. Here is the sample code.</p>
<pre><code class="lang-auto">import torch

n = torch.randn(200, 385)
s = n @ n.T
print(torch.allclose(s, s.T))
</code></pre>
<p>This prints False.</p>
<p>Is this expected behavior? Shouldn’t it return True? What am I missing here? Is this a bug?</p>

Joschka: <p>Seems like an issue with <a href="https://pytorch.org/docs/stable/notes/numerical_accuracy.html" rel="noopener nofollow ugc">numerical accuracy</a>. If this is a problem at some point in your code you can either</p>
<ol>
<li>Increase numerical precision, i.e., in this case <code>n = torch.randn(200, 385, dtype=torch.float64)</code>.</li>
<li>or do the equality check with higher tolerance, e.g., <code>torch.allclose(s, s.T, atol=1e-4)</code>. Note that here the default is <code>atol=1e-8</code> and the “a” stands for tolerance with respect to <strong>a</strong>bsolute deviation.</li>
</ol>

alywonder: <p>Thank you. I was hoping that if I used default type for tensor and default values when I call allclose(), the behaviors would be coherent.</p>
A Differentiable Way for Transforming From Logits to Binary

patatessever: <p>Hi, I have a semantic segmentation network. At some point, I want to define a loss function that calculates the boundaries in the predicted image. Even though the task is for binary images, the output of the network is in logits. Is there any differentiable way to transform the predicted mask from logits to a binary image? I tried the softargmax but it did not help.<br>
Moreover, in this paper “A Novel Boundary Loss Function in Deep Convolutional Networks to Improve the Buildings Extraction From High-Resolution Remote Sensing Images” they propose a binarization function but I don’t understand how it changes the dimensions. Can anyone help?</p>

tataganesh: <p>Do you want the segmentation network to output a mask with two classes - Border and background?</p>

patatessever: <p>No my prediction will be in the building and background, I want to use boundaries in the loss function</p>

KFrank: <p>Hi Yunus!</p>
<aside class="quote no-group" data-username="patatessever" data-post="1" data-topic="174080" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/patatessever/48/57624_2.png" class="avatar"> patatessever:</div>
<blockquote>
<p>I want to define a loss function that calculates the boundaries in the predicted image. Even though the task is for binary images, the output of the network is in logits. Is there any differentiable way to transform the predicted mask from logits to a binary image?</p>
</blockquote>
</aside>
<p>There is no usefully differentiable way to map to a binary (or any discrete)<br>
result.  For the range of input values that map to <code>0</code>, the derivative will be<br>
zero.  Right when you jump to <code>1</code>, the derivative will be technically undefined.<br>
Then for the range that maps to <code>1</code>, the derivative will be zero again.</p>
<p>So even though well defined almost everywhere, the derivative provides<br>
no useful information about how to minimize your loss function with gradient<br>
descent.</p>
<p>In general, you would want to figure out how to use the logits directly in your<br>
loss function.  (<code>BCEWithLogitsLoss</code> does so quite effectively for binary<br>
classification problems.)</p>
<p>If you must (but you probably don’t want to for reasons of numerical stability),<br>
you can convert the logits to probabilities by passing them through <code>sigmoid()</code>.</p>
<p>(Another approach – that I don’t recommend – would be to binarize your<br>
logits, but then use surrogate derivatives that approximate your binarization<br>
derivatives and that somehow capture the information you need to be able<br>
to perform gradient descent.)</p>
<p>Without knowing what your loss function looks like, it’s hard to give concrete<br>
advice.</p>
<p>Best.</p>
<p>K. Frank</p>
Pre-trained Entity Embeddings

Andreas_Binder: <p>Hi all! I am looking for pre-trained entity embeddings <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> I know that PyTorch provides word embeddings like GloVe, thus I was curious if there are pre-trained entity embeddings too!<br>
Thank you!</p>

AbdulsalamBande: <p>I am not sure what you mean by pre-trained entity embeddings, but this site <a href="https://huggingface.co/dslim/bert-base-NER/tree/main" rel="noopener nofollow ugc">Hugging Face</a> has a model for predicting entities, you can extract the weights of the embedding layer.</p>

Andreas_Binder: <p>Awesome, thanks! Sorry for the confusion, I thought there are also non-transformer based embeddings for entities as there are for just words (eg GloVe).</p>
How to interpret results from CocoEvaluator

Joysn: <p>I am using the engine.py stuff which uses CocoEvaluator. I tried to get information about CocoEvaluator and tried to understand the code, but I am not sure how to interpret the figures from the output, nor how they relate to the mAP values which are mentioned in all papers.<br>
If a model performs good, are the average precision and recall values close to 1? And what does -1 mean?</p>
<pre><code class="lang-auto">IoU metric: bbox
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.337
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.391
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.381
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.649
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.326
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.048
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.213
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.837
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.680
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.842
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
IoU metric: segm
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.324
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.391
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.387
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.475
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.318
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.045
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.201
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.820
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.740
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.822
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
</code></pre>

anmorgan24: <p>I know this post is old, but in case anyone else is looking for an answer to this, I found this link to be very helpful: <a href="https://cocodataset.org/#detection-eval" class="inline-onebox" rel="noopener nofollow ugc">COCO - Common Objects in Context</a></p>

Joysn: <p>Hello, thank you. I used the COCO documentation too and got more details about the results. I could even extract data for my own evaluation after looking at the code of CocoEvaluator.</p>
Check filter type in convolutional layer CNN

Anastasia_Berlianna: <p>i build unet encoder with pretrained resnet34 with weight from imagenet</p>
<pre><code class="lang-auto">resnet = models.resnet34(weights=models.resnet.ResNet34_Weights.IMAGENET1K_V1)

class Encoder(nn.Module):
    def __init__(self):
        super().__init__()
        self.layer1 = nn.Sequential(
            resnet.conv1,
            resnet.bn1,
            nn.LeakyReLU(negative_slope=0.1, inplace=True),
            resnet.layer1
        )
        self.layer2 = resnet.layer2
        self.layer3 = resnet.layer3
        self.layer4 = resnet.layer4
    
    def forward(self, x):
        x1 = self.layer1(x)
        x2 = self.layer2(x1)
        x3 = self.layer3(x2)
        x4 = self.layer4(x3)
        return x1, x2, x3, x4
</code></pre>
<p>i want to check the filters type used in each convolutional layer (e.g like sobel, prewitt, gaussian, etc). How can i check that?</p>

ptrblck: <p>The filters of conv layers are trained and while they might learn common image processing filters there won’t be a guarantee they will.<br>
You can access the filters via the <code>.weight</code> attribute in each conv layer and could then try to visualize it.</p>

Anastasia_Berlianna: <p>Thank you! Is there any difference if i acces the weight before or after training? And how do i visualize it?</p>

ptrblck: <p>Yes, there will be a difference since the <code>weigth</code> attribute, which represents the filters, is trained.<br>
You could visualize each input channel of each filter using e.g.:</p>
<pre data-code-wrap="python"><code class="lang-python">conv = nn.Conv2d(in_channels=8, out_channels=16, kernel_size=3)
out = make_grid(conv.weight.view(-1, 1, conv.weight.size(2), conv.weight.size(3)), nrow=conv.out_channels, normalize=True)
plt.imshow(out.permute(1, 2, 0).numpy())
</code></pre>
<p>Output:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/f/3/f3812ebb81e46722ef253f34b0bb09ac391126d1.png" alt="image" data-base62-sha1="yK8Nsb4o52A18MyJrfmZo3Sqr73" width="368" height="204"></p>

Anastasia_Berlianna: <p>ah get it, thank you!</p>
Cross Entropy Loss different after expanding last dimension to VOCAB_SIZE

alessandro_mondin: <p>Hello all,<br>
I need to run Cross Entropy on a vocabulary prediction and I noticed that there is no consistency between the result of the suggestion by king Patrick <a href="https://discuss.pytorch.org/t/how-do-i-use-nn-crossentropyloss-for-seq2seq-where-my-predication-is-of-size-bs-seq-len-vocab-size-and-truth-of-size-bs-seq-len/80527">here</a> and my implementation that with a trivial &amp; straightforward trick increases the last dimension to VOCAB_SIZE. Here below the code:</p>
<pre><code class="lang-auto">    def training_step(self, batch, batch_idx):
        img, question, answer = batch
        y_hat = F.softmax(self(img, question, answer), dim=-1)

        # PROBABLY RIGHT
        loss_suggested_by_ptrblck = self.loss(y_hat.permute(0, 2, 1), answer) # -&gt; tensor(10.8249, grad_fn=&lt;NllLoss2DBackward0&gt;)
        # y_hat.permute(0, 2, 1) --&gt; torch.Size([8, 50265, 5]) --&gt; bs, vocab_size, seq_len
        # answer.shape --&gt; torch.Size([8, 5]) --&gt; bs, seq_len


        answer_logits = torch.zeros_like(y_hat)
        answer_logits = answer_logits.view(answer_logits.size(0) * answer_logits.size(1), -1)
        answer_logits[torch.arange(answer_logits.size(0)), answer.flatten()] = 1.0
        answer_logits = answer_logits.view(y_hat.size(0), -1, answer_logits.size(-1))

        assert torch.equal(torch.argmax(answer_logits, dim=-1), answer)
                                           
        loss = self.loss(y_hat, answer_logits) # tensor(0.0007, grad_fn=&lt;DivBackward1&gt;)
        # y_hat.shape --&gt; torch.Size([8, 5, 50265]) --&gt; bs, seq_len, vocab_size
        # answer_logits.shape --&gt; torch.Size([8, 5, 50265]) --&gt; bs, seq_len, vocab_size

        return loss
</code></pre>
<p>As you can see mine is ~ 0.0007 while Patrick’s is ~ 10.8<br>
How is it possible?<br>
Thanks a lot in advance</p>