akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
State of afairs for development w/ remote GPUs as of 2024

matheusd: <p>Hello,</p>
<p>I’m a PyTorch newbie and I’m trying to assess whether it’s <em>possible</em>/<em>feasible</em>/<em>reliable enough</em> to develop with a remote GPU in 2024.</p>
<p>The latest information I have found so far is the following post in the forum, which would seem to imply it is: <a href="https://discuss.pytorch.org/t/ease-development-by-running-computations-on-remote-gpu/121002/7" class="inline-onebox">Ease development by running computations on remote GPU - #7 by wayi</a></p>
<p>However, I want to to highlight the following quotes:</p>
<blockquote>
<blockquote>
<p>Documentation on RemoteModule says RemoteModule is not currently supported when using CUDA tensors, but you said tensors will be automatically placed to the same cuda device. Am I missing something? If CUDA tensors are not supported now, where can I track progress on this?</p>
</blockquote>
</blockquote>
<blockquote>
<p>Thanks for pointing this out! The doc is outdated. Actually CUDA tensors are now supported on TensorPipe backend, documented on the same page. I will update the doc soon.</p>
</blockquote>
<p>Notably, the documentation page  <em>still</em> has this warning, almost 3 years later.</p>
<p>And as of today (Jan/2024) it’s actually worse, because the <a href="https://github.com/pytorch/tensorpipe" rel="noopener nofollow ugc">TensorPipe backend</a> repository is archived without any notices or otherwise hints about whether it’s still supported, if development has moved somewhere else or if it has been superceded.</p>
<p>The original issue (pytorch issues 33979) about supporting tensor pipe is open, and has had no updates since it’s been created.</p>
<p>I also tried eyeballing through the first few among the 300+ issues and 1k+ PRs with a reference to <code>tensorpipe</code>, but the hits on the search seem to be overwhelmingly about support issues related to compilation errors.</p>
<p>My question is: is running functions on remote CUDA devices possible, in 2024? Is there an <em>updated</em>, <em>authoritative</em> source or venue I can rely on to give me this information?</p>
Question about QAT

BRH: <p>Apologies, I am new to quantization.<br>
The current QAT tutorial on the official website runs the training entirely on CPU.<br>
I realized the QAT model can be trained on GPU before calling torch.ao.quantization.convert on the model.</p>
<p>Will training the QAT model in GPU mode affect anything?</p>

jerryzh168: <p>it will use cuda kernel instead of cpu kernel, this includes original model operators like batchnorm and the fake quantize operators that’s added by quantization flow, typically it won’t have much impact on accuracy.</p>
I have a question about tensor.to(non_blocking=True)

woongjoonchoi: <p>I saw topics on <a href="https://discuss.pytorch.org/t/should-we-set-non-blocking-to-true/38234/11">[Should we set non_blocking to True? - PyTorch Forums]</a><br>
. I read answers and i saw post [[How to Overlap Data Transfers in CUDA C/C++ | NVIDIA Technical Blog]]   <a href="http://developer.nvidia.com/blog/how-overlap-data-transfers-cuda-cc/" class="inline-onebox" rel="noopener nofollow ugc">How to Overlap Data Transfers in CUDA C/C++ | NVIDIA Technical Blog</a>   (add https:// i’m new user so , i can’t add another link)</p>
<p>I wonder if my understanding is correct .</p>
<p>Following code is from ptrblck’s answers.</p>
<pre data-code-wrap="python"><code class="lang-python">for data, target in loader:
    # Overlapping transfer if pinned memory
    data = data.to('cuda:0', non_blocking=True)
    target = target.to('cuda:0', non_blocking=True)

    # The following code will be called asynchronously,
    # such that the kernel will be launched and returns control 
    # to the CPU thread before the kernel has actually begun executing
    output = model(data)  # has to wait for data to be pushed onto device (synch point)
    loss = criterion(output, target)
    loss.backward()
    optimizer.step()

</code></pre>
<p><img src="https://developer-blogs.nvidia.com/wp-content/uploads/2012/11/C2050Timeline-1024x670.png" alt="C2050Timeline-1024x670.png (1024×670) (nvidia.com)" width="690" height="451"></p>
<p>i assume that async data transfer to gpu is available . Also, i use single gpu.</p>
<p>Q. Is the data splitted into small chunk and operation executed like async version 2 in picture?</p>
<p>From what I understand  ,  batch_data is splitted in small chunk (nvidia’s async transfer). Then , kernel(in this code, <code>output = model(data) ,)loss = criterion(output, target) is executed .    </code>   loss.backward() optimizer.step()`  i think backward is computed for batch_data if there is a batchnormalization . So ,In my opinion, backward   should be sync point .<br>
Also, I thinnk that optimize.step must be blocked to main process. Because, model must compute output after model parameter’s update.<br>
What i undersatnd is correct?</p>
<p>I have another question<br>
Q .<br>
When GPu execute ouput calculate, loss calculate , backward , optimize by gpu  ,  Cpu execute data transfer to gpu  parallely(<code> data = data.to('cuda:0', non_blocking=True)     target = target.to('cuda:0', non_blocking=True)</code> ) ?</p>
<p>In above picture, there are kernel engine and h2d engine. From what i understand, Kernel engine and h2d engine are separate. h2d engine just transfer data to device regardless of kernel engine. Kernel Engine wait until optimization step finish. Then , If data  has been transferred to device already, Kernel Engine computes model ouputs. In Conclusion , DataLoaderPart(data transfer to gpu) and ModelComputationpart(ouput,loss,backward,optimize) are executed paralelly , but model output can be computed when data has been transferred to device already.</p>
<p>Thank you for seeing my topic</p>
My code seems to stop while moving tensor to GPU?

miturian: <p>Occasionally, my code will seem to ‘hang’ indefinitely at some point during training, with no errors or anything indicating a problem. Going into task manager, ‘python’ still seems to be taking up both cpu and gpu resources. Besides the lack of progress in training, the only sign that something is broken is that if I try to stop the code, spyder becomes unresponsive. At first I thought it might be spyder, but when I run the coder directly from the anaconda prompt, I can still get the same behavior (except that the anaconda prompt will allow me to ctrl-C the execution).<br>
After placing a lot of ‘print’'s around the training loop, it seems that the code stops while executing these lines:</p>
<pre><code>    xbatch=xtemp.to(device=cuda)
    ybatch=ytemp.to(cuda)
</code></pre>
<p>I see a lot of people with ‘hanging’ code gets told to look at their data loading, but all my data is in memory (and the dataloader has num_workers=0). And these lines are after the dataloader has already spit out xtemp and ytemp…</p>
<p>How do I debug this?</p>
<p>Before I kill it, the gpu stats in taskmanager looks like this:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/6/6/6698cf0b1e508a258421007997f8dd8e4bc5685f.png" alt="Screenshot 2021-03-01 142136" data-base62-sha1="eDC5MsrfpZSma77bnE3UOVc4gtp" width="333" height="202"></p>
<p>So, it’s not immediately clear that the GPU is overworked…</p>
<p>After I kill the script from the anaconda prompt, all I get is:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/d/1d03d54c444da55b0a3550d4938dea2571b5f858.png" data-download-href="https://discuss.pytorch.org/uploads/default/1d03d54c444da55b0a3550d4938dea2571b5f858" title="prompt"><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/d/1d03d54c444da55b0a3550d4938dea2571b5f858.png" alt="prompt" data-base62-sha1="48G3FF4zjurgXrY2LVBRffVpjbW" width="690" height="129" data-dominant-color="1F1F1F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">prompt</span><span class="informations">692×130 4.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Yunxuan_Xiao: <p>Any luck? I also encountered the similar issue.</p>
Gradients disabled globally?

some_user1: <p>Hi, so I have this very basic and unrealistic example:</p>
<pre><code class="lang-auto">a = torch.linspace(-1,1,100).reshape(-1,1)
b = a

c = torch.nn.Sequential(
    nn.Linear(1,10),
    nn.ReLU(),
    nn.Linear(10,1)
)

opt = torch.optim.Adam(c.parameters(), lr=1e-3)

for i in range(10):
  opt.zero_grad()
  b_hat = c(a)
  loss = F.mse_loss(b_hat,b)
  loss.backward()
  opt.step()
</code></pre>
<p>So sure, it’s not a realistic training, however it should nevertheless work (in the sense that there shouldn’t be any error). Yet I have this:</p>
<pre><code class="lang-auto">RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
</code></pre>
<p>Which is caused by the line loss.backward().</p>
<p>I cannot understand why this happens. If any one can help, I’d be gratefull.</p>
<p>For context, I run this on colab. I ran the exact same code on my terminal and it worked. I ran a similar code in colab, and it worked. Why would the gradients be disabled globally in this peculiar environment ?</p>

ptrblck: <p>Your code works fine for me so you might indeed have disabled gradient calculation in the failing script e.g. via <code>torch.set_grad_enabled(False)</code>.</p>

some_user1: <p>Thank you very much for your answer. I thought about it and I was planning on putting a torch.set_grad_enabled(True) (because nowhere in my code did I ever disable the grad except in torch.no_grad() blocks, but this error happened outside these blocks).</p>
<p>I just re-ran all the code and now magically it works !</p>
<p>However I stumbled upon this :</p>
<pre><code class="lang-auto">&lt;ipython-input-11-46a46f132a51&gt;:1: UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the .grad field to be populated for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more informations. (Triggered internally at aten/src/ATen/core/TensorBody.h:489.)
</code></pre>
<p>This error came while running the following function</p>
<pre><code class="lang-auto">def infer(x):
  for layer in m: # m is a torch.nn.ModuleList
    x = layer(x)
  return x

infer(x).grad
&gt;&gt; None
</code></pre>
<p>It’s just a warning but however I don’t quite understand why it appears.</p>
Filter Pruning - Remove zero filters

Konstantinos_Gkrispa: <p>Hi, I want advice on net pruning. I have implemented pruninig to an object detector with FPN and skip connections using NNI’s library. The problem is that NNI’s ModelSpeedup() isn’t compatible with my model’s architecture and I am left with a model with zero filters. I want to remove those filters…</p>
<p>Is there any tool or any way to permantly remove those zero filters and not mess with the model ?</p>
Is anyone available to assist me in resolving an error? I'm new to this topic, and the code I'm working with utilizes Opacus Version 1.4.0. Your guidance and support would be greatly appreciated

Rafika_Benledghem: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/9/09bb9fe550518aacaccbd069ddde8da8f77a4f97.png" data-download-href="https://discuss.pytorch.org/uploads/default/09bb9fe550518aacaccbd069ddde8da8f77a4f97" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/9/09bb9fe550518aacaccbd069ddde8da8f77a4f97.png" alt="image" data-base62-sha1="1o6hqqTvwdXaaulduodzYB1HHyn" width="690" height="220" data-dominant-color="3B3B3C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1248×398 25.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
How can Pytorch 2.x support different backends?

jiewang: <p>How does Pytorch 2.x support backend hardware from different vendors like Intel, NVIDIA, and AMD? PyTorch 1.x appears to necessitate individual vendors to develop operators that are compatible with their specific platforms.</p>

marksaroufim: <p>For CPU Intel wrote some custom codegen that outputs C++ code<br>
For GPU like vendors we codegen triton kernels and those are increasingly becoming supported by more hardware vendors so there’s nothing pytorch really needs to do to support more GPU like vendors as long as they integrate with triton</p>

jiewang: <p>I got it. Thanks for your reply!</p>
Integrated Gradients and Text Generation

jakobamb: <p>Hi all,</p>
<p>I am currently trying to explain the generated Text of a GPT-2 model, using Integrated Gradients and captum. I can create a minimal example if helpful.</p>
<p>My question is: Is it even at all possible to explain text generation? My current approach is to explain each token individually, by viewing the logits of the output layer as a classification problem and varying the context the model receives. As the target, I assigned the token index that is generated without pertubation. However, the attributions to the input are all zero. Do I make a conceptual mistake here? Is there a resource somewhere for explaining text generation? I couldn’t seem to find anything…</p>

arotte: <p>I’m having a really similar problem. Did you manage to find anything on the topic?</p>

jakobamb: <p>The way I solved it was basically what I had written above (I am assuming an autoregressive text generation model like GPT here):</p>
<ul>
<li>run the model to generate one token, use the index of that token as your target for the token-classification output</li>
<li>calculate the attribution for that token</li>
<li>add the generated token to your input and repeat the two steps above to calculate the attributions for the next token</li>
</ul>
<p>That will give you an attribution for every token that was generated and you bascially just loop autoregressively until your text generation is finished. It is then up to you what you do with those attributions, but you could for example calculate the attributions only for the prompt and in the end average the attributions that you calculated for each token to get an attribution for the whole generated text. Note that in that case, you have to calculate attributions only for your initial prompt then, you can feed in the generated tokens using additional forward args if you are using captum.</p>
<p>Hope this helps and is understandable, if not I can prepare a python snippet, unfortunately my code became a bit more complicated to use it as an example…</p>

MisterXY89: <p>A bit late to the party, but I would be highly interested in how you solved this code-wise!</p>
How can I save the model as pt file with specific classes?

jjoon0928: <p>I am currently studying deep metric learning. I use Cars-196 dataset and save the model in each iteration or epoch to check how data and proxies are updated. But there are quite many classes so I think that it’d be better to reduce classes, save and visualize them.</p>
<p>What is better way to modify for saving specific classes?</p>
<ol>
<li>reduce the model before training: it means, for example, train only 10 classes</li>
<li>save the model after training but only save 10 specific classes: train all the 98 classes and just save 10 specific classes</li>
<li>save all the data but visualize only 10 specific classes.</li>
</ol>

marksaroufim: <p>It’s generally a good idea to start with fewer classes as you build up intuition on the problem, your iteration cycle will be faster</p>
<p>Typically datasets like Cars will have the labels be part of the folder structure or name of the file or some extra metadata so filter out all but 10 classes. Train a model for those 10 classes and see if you can get your model to overfit. Then increase the number of classes and build a better model <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

jjoon0928: <p>Yes your Right. I thought it overcomplicating. I just set 10 classes train and set others as test. Thank you so much!</p>
Select the right observers in QAT

xjtuljy: <p>Hi, I’m playing with different observers, by defining weight and activation observers in torchvision ImageNet classification (<a href="https://github.com/pytorch/vision/blob/6a9ac7271beed04fb85cfa99506618cdde36a48c/references/classification/train_quantization.py#L58C16-L58C16" rel="noopener nofollow ugc">link</a>) using:<br>
<code>torch.ao.quantization.QConfig()</code><br>
I found that by changing from MovingAverageMinMaxObserver to other observers like MinMaxObserver leads to a performance drop from QAT FP model to quantized INT8 model.</p>
<p>Is there any guidance regarding observer selection for QAT in practice?</p>

HDCharles: <p>QAT is a training process, its better to have an activation observer that changes as the model changes and doesn’t have an infinite memory. MovingAverageObservers have this, other observers do not.</p>
<p>If your weights shift such that your activations originally were -1 to 1 and now go from -.1 to .1, your minmaxobserver will still try to handle values from -1 to 1, which is not what you want.</p>

xjtuljy: <p>Thanks for the explanation! Can we then say MovingAverageObservers is in general preferred over Minmaxobserver? I’ll read the source code to get better understanding.</p>

HDCharles: <p>Yeah, for activations while doing QAT you usually want moving average observer. For weights you usually do a moving average observer with the constant set so it has no memory. For PTQ it’s usually histogram observer for activations and minmaxobserver for weights</p>

xjtuljy: <p>I see. I have two follow-up questions:</p>
<ol>
<li>
<p>why can there be a gap between QAT FP model and quantized INT8 model? My understanding towards QAT is that, it applies constraints to make the FP model to have same weights as INT8 model, so technically it should yield exactly the same/very close results to the quantized model, regardless of what observer is used. Is my understanding correct?</p>
</li>
<li>
<p>for “moving average observer with the constant set” do you mean setting “quant_min” and “quant_max”, or “self.min_val” and “self.max_val” to fixed values?</p>
</li>
</ol>

HDCharles: <p>I’m not sure what you mean. There isn’t just one quantized model. Doing PTQ vs QAT yields different models, otherwise there’d be no point to do QAT.</p>
<p>No, those contestants don’t control the memory/momentum</p>
How to use multiple LSTM?

hirundo: <p>Hello.<br>
I want to create model like the picture.<br>
Is it okay to write code like this?</p>
<pre><code>def __init__(self):
    super(Feature_extracter, self).__init__()
    self.lstm = nn.LSTM(input_dim, 50, batch_first=True)
    self.lstm2= nn.LSTM(50, 50, batch_first=True)
    self.lstm3= nn.LSTM(50, 20, batch_first=True)
    self.dropout = nn.Dropout(0.2)

def forward(self, input):
    h0 = torch.zeros(1,1,50)
    c0 = torch.zeros(1,1,50)
    output, (h, c) = self.lstm(input, (h0, c0))
    output  = self.dropout(output)
    output, (h, c) = self.lstm2(output, (h, c))
    output  = self.dropout(output)
    output, (h, c) = self.lstm3(output, (h, c))
    output = self.dropout(output)
</code></pre>
<p>Please help a newbie developer <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/8/8866f6c75355b60cd762730f4126287c23753f40.png" alt="image" data-base62-sha1="jsFxbEgp44lGJ7trxhZVILn5dxC" width="425" height="152"></p>

J_Johnson: <p><code>nn.LSTM</code> has a <code>num_layers</code> argument. You could just set that to 3. It also has a <code>dropout</code> argument, which just specifies the p value for dropout applied to each layer.</p>

hirundo: <p>I love you ! Thank you ~~</p>
How to infer torch model in scala spark?

Danfeng_Zhang: <p>We train a model using pytorch, and need to infer it in scala spark tasks with cpu.However, I do not find any resources about it.<br>
torch version :1.7.1</p>
<p>Thank you</p>

marksaroufim: <p>It’s unlikely you’ll find spark experts on this forum but you can ask the spark folks here <a href="https://stackoverflow.com/questions/tagged/apache-spark" class="inline-onebox">Newest 'apache-spark' Questions - Stack Overflow</a></p>
<p>And i found this somewhat recent blog post on databricks website <a href="https://www.databricks.com/blog/2023/04/20/pytorch-databricks-introducing-spark-pytorch-distributor.html" class="inline-onebox">PyTorch on Databricks - Introducing the Spark PyTorch Distributor</a></p>
How to save a wrapped transformer model using torch.script.trace?

chrxu: <p>I have a custom method that wraps around a sequence classification model and I’m having trouble saving the wrapped model using torch.jit.trace</p>
<p>When I run the following line, I get this ambiguous runtime error:</p>
<pre><code class="lang-auto">torch.jit.trace(IGWrapper(model=model), (input_ids, attention_mask, baseline))
</code></pre>
<p>RuntimeError: 0 INTERNAL ASSERT FAILED at “…/torch/csrc/jit/ir/alias_analysis.cpp”:616, please report a bug to PyTorch. We don’t have an op for aten::select but it isn’t a special case.  Argument types: Tensor?, int, int,</p>
<p>Candidates:<br>
aten::select.Dimname(Tensor(a) self, str dim, int index) → Tensor(a)<br>
aten::select.int(Tensor(a) self, int dim, SymInt index) → Tensor(a)<br>
aten::select.t(t<a></a> list, int idx) → t(*)</p>
<p>Here is my method for context:</p>
<pre><code class="lang-auto">@torch.jit.script
def calc_integral(grads):
    # approx integral
    grads = (grads[:-1] + grads[1:]) / 2
    avg_grad = grads.mean(0)
    return avg_grad

class IGWrapper(nn.Module):
  def __init__(self, model):
    super().__init__()
    self.model = model.eval()

  def forward(self, 
              input_ids: Tensor, 
              attention_mask: Tensor,
              baseline: Tensor):
    torch.set_grad_enabled(True)
    input_embed = self.model.base_model.embeddings.word_embeddings(input_ids)

    copy_embed = torch.clone(input_embed)

    if baseline is None:
    # create baseline
      baseline = torch.zeros_like(copy_embed)

    grads = []

    num_steps = 5
    for step in range(num_steps + 1):
      print(f"step: {step}/{num_steps}")
      input_embed.data = baseline + step/num_steps * (copy_embed - baseline)
      outputs = self.model(input_ids, attention_mask, output_hidden_states=True, output_attentions=True)

      logits, hidden_states = outputs.logits, outputs.hidden_states

      # calculate the derivates of the output embeddings
      out_embed = hidden_states[0]

      @torch.jit.script
      def grad_fn(logits: Tensor, out_embed: Tensor) -&gt; Optional[Tensor]:
          grad_outputs: List[Optional[torch.Tensor]] = [torch.ones_like(logits)]
          return torch.autograd.grad([logits], [out_embed], grad_outputs=grad_outputs, create_graph=True)[0]
      
      g = grad_fn(logits, out_embed)

      grads.append(g[0])

    # stack grads along first dimension to create a new tensor
    grads = torch.stack(grads)

    avg_grad = calc_integral(grads)

    integrated_grads = out_embed * avg_grad

    scores = torch.sqrt((integrated_grads ** 2).sum(-1))

    # normalize scores
    max_s, min_s = scores.max(1, True).values, scores.min(1, True).values

    normalized_scores = (scores - min_s) / (max_s - min_s)
    return normalized_scores[0].detach()
    
</code></pre>

marksaroufim: <p>FYI <code>torch.jit</code> is under maintenance mode so recommendation is to switch to using <code>torch.export</code> if you want no python or <code>torch.compile()</code> if you just care about speedups</p>
Dark theme for entire PyTorch website

Nav9: <p>I see a discussion <a href="https://discuss.pytorch.org/t/documentation-theme-0-4-1-vs-1-0-0/26315">here</a> along similar lines. I know there are bigger fish to fry, but if it would be possible to get this implemented by perhaps assigning the task to an intern, it’d be a big plus.<br>
Basically, it’d be nice to have a toggle button on every page, which allows a user to switch the website to a dark theme. It’s quite tiring on the eyes, to stare at a white background.<br>
I noticed this forum has a dark mode option in preferences. But I’m referring to having it on the PyTorch website, including the tutorial pages.</p>

ptrblck: <p>CC <a class="mention" href="/u/suraj.pt">@suraj.pt</a> for viz</p>

Julius_Smith: <p>This would be great.  My workaround is to use the “Dark Mode Chrome” extension in Chrome, along with a Chrome dark-mode theme.</p>

marksaroufim: <p>This is an area where contributions would be very much welcome - if anyone can get it working here <a href="https://github.com/pytorch/pytorch_sphinx_theme" class="inline-onebox">GitHub - pytorch/pytorch_sphinx_theme: PyTorch Sphinx Theme</a> feel free to tag <a class="mention" href="/u/svekars">@svekars</a> and me on github</p>
The memory footprint grow irregularly with the increased of vision prompts

524048328: <p>I hope this message finds you well. I am currently exploring the shallow VPT method as proposed in <a href="https://github.com/KMnP/vpt/blob/main/src/models/vit_prompt/vit.py" rel="noopener nofollow ugc">this GitHub link</a> and would like to experiment with different numbers of inserted prompts. Interestingly, I have observed that the peak memory footprint grows irregularly with the increase of visual prompts, with no significant increase from 2 to 3. I have shared the results below. I’m curious to understand which features in PyTorch contribute to this outcome. Your insights would be greatly appreciated.</p>
<div class="md-table">
<table>
<thead>
<tr>
<th>tokens number</th>
<th>0</th>
<th>1</th>
<th>2</th>
<th>3</th>
<th>4</th>
<th>5</th>
<th>6</th>
<th>7</th>
<th>8</th>
<th>9</th>
<th>10</th>
</tr>
</thead>
<tbody>
<tr>
<td>Memory (MB)</td>
<td>818.9</td>
<td>824.1</td>
<td>827.1</td>
<td>827.1</td>
<td>828.6</td>
<td>831.1</td>
<td>831.9</td>
<td>836.1</td>
<td>836.9</td>
<td>838.6</td>
<td>842.4</td>
</tr>
<tr>
<td>Extra (MB)</td>
<td>0</td>
<td>5.2</td>
<td>8.2</td>
<td>8.2</td>
<td>9.7</td>
<td>12.2</td>
<td>13</td>
<td>17.2</td>
<td>18</td>
<td>19.7</td>
<td>23.5</td>
</tr>
</tbody>
</table>
</div>

marksaroufim: <p>Not an expert in this specifically but typically the CUDA caching allocator will sometimes request more memory than it needs and manage that memory <a href="https://zdevito.github.io/2022/08/04/cuda-caching-allocator.html" class="inline-onebox">A guide to PyTorch’s CUDA Caching Allocator | Zach’s Blog</a></p>
Fast L1 all-to-all distance

wiiiktor: <p>Attention mechanism is using a dot product, which is fast (tensor1 @ tensor2) multiplication. Is there also a fast L1 distance (tensor1 L1 tensor2) function? // L1 is just a subtraction and I am surprised that I can not find an implementation faster than multiplication…</p>

wiiiktor: <p>Anyone?.. (and some more characters, to reach 20)</p>
How to calculate FLOPS of a JIT model?

JimW: <p>I was using <code>from fvcore.nn import FlopCountAnalysis</code> to calculate the FLOPS for PyTorch model, and it worked very well.</p>
<p>But this time, I received a JIT model from someone else and I loaded it using <code>network = torch.jit.load("model.pth")</code>.  It seems <code>FlopCountAnalysis</code> will crash this time with the following error messages:</p>
<blockquote>
<p>File “/lib/python3.8/site-packages/fvcore/nn/jit_analysis.py”, line 248, in total<br>
stats = self._analyze()<br>
File “/lib/python3.8/site-packages/fvcore/nn/jit_analysis.py”, line 551, in _analyze<br>
graph = _get_scoped_trace_graph(self._model, self._inputs, self._aliases)<br>
File “/lib/python3.8/site-packages/fvcore/nn/jit_analysis.py”, line 174, in _get_scoped_trace_graph<br>
register_hooks(mod, name)<br>
File “/lib/python3.8/site-packages/fvcore/nn/jit_analysis.py”, line 157, in register_hooks<br>
prehook = mod.register_forward_pre_hook(ScopePushHook(name))<br>
File “/lib/python3.8/site-packages/torch/jit/_script.py”, line 942, in fail<br>
raise RuntimeError(name + " is not supported on ScriptModules")<br>
RuntimeError: register_forward_pre_hook is not supported on ScriptModules</p>
</blockquote>
<p>I know it must be due to the difference between a normal PyTorch model and a JIT loaded model.  Could someone please suggest some method to evaluate the FLOPS of this model?  Thanks.</p>

marksaroufim: <p>AFAIK this is not super straightforward, the way fvcore works is that it’ll insert some hooks that will observe what an <code>nn.Module</code> is doing to count its FLOPS. <code>torch.jit</code> is under maintenance mode so even if it was possible to support hooks it’s unlikely to get prioritized and its not trivial to convert from a JIT model to a regular PyTorch model so your best bet is to profile the model without jit</p>
Where to PrimTorch's operator list?

AidanGoldfarb: <p>I have read about Torch’s <a href="https://pytorch.org/get-started/pytorch-2.0/#technology-overview" rel="noopener nofollow ugc">primitive operations</a> in several places, including blog posts, but I have never seen a full list of the ~250 operations. I am doing some low level code analysis and a list like this would be useful. Any pointers would be greatly appreciated</p>
<p>Thanks</p>

marksaroufim: <p>This is probably your best bet <a href="https://pytorch.org/docs/master/ir.html" class="inline-onebox">IRs — PyTorch master documentation</a></p>

AidanGoldfarb: <p>Ah that will suffice, thank you! I had seen that page once before, but could not find it again. However, there are only about 120 operators there, is the ~250 number no longer accurate?</p>
How to Create a 1D Embedding from Tensors of Varying Sizes?

pkamranjam: <p>I am a newbie in AI and playing with some computer vision algorithms.</p>
<p>I have three tensors with different sizes. Noise augmentation levels tensor with size (N, C, H, W), diffusion timestep tensor of size (N, H) and pooled pose embeddings of size (N, C, H, W). I need to sum these tensors so that the 1D embedding result can be fed to FiLM layer.</p>
<p>How can I apply the summation without losing data?</p>
<p>Thank you!</p>