akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
CUDA memory issue probably by Python

M2M: <p>Hello everyone, I need your kind help.</p>
<p>I have to run my U-net model on a large 3D image, so I chop it into smaller portions, run it individually, and then put them together.</p>
<p>code_1.py chops the large 3D image into smaller portions and calls code_2.py inside a for loop where each call contains a small piece of the (chopped) image.</p>
<p>Everything works fine when I call code_2.py inside code_1.py as:</p>
<pre><code class="lang-auto">for small_img in large_img_list:
	torch.cuda.empty_cache()
	os.system('%s %s %s' % ('python3', 'code_2.py', small_img))
</code></pre>
<p>However, when I put the code of code_2.py into a function inside the same file (code_2.py) and import this file into code_1.py and do function calls, I run out of CUDA memory. The code is below.</p>
<pre><code class="lang-auto"># This is code_1.py
from code_2 import *

for small_img in large_img_list:
	torch.cuda.empty_cache()
    RunModels(small_img)
</code></pre>
<p>Why is this? I clear every tensor I allocate inside code_2.py</p>
<pre><code class="lang-auto">torch.cuda.empty_cache()
del image
del mask
</code></pre>
<p>Maybe this is a python problem rather than PyTorch problem?</p>

ptrblck: <p>Your import might be already executing some code and maybe even initializing a (separate) CUDA context. You could check if my adding <code>print</code> statements to the methods on <code>code_2</code> and also check the memory usage right after the <code>import</code>.</p>

M2M: <p>Thank you for your answer. You are right, when I import <em>code_2.py</em>, it does execute lines in the global context inside <em>code_2.py</em> which is understandable but it doesn’t obviously execute the function <code>RunModels()</code> which is inside <em>code_2.py</em> and doing all the work.</p>
<p>However, when I print the memory allocation, it doesn’t allocate any memory right before and after import.</p>
<pre><code class="lang-auto">t = torch.cuda.get_device_properties(0).total_memory
r = torch.cuda.memory_reserved(0)
a = torch.cuda.memory_allocated(0)
f = r-a  # free inside reserved

print("\ttotal_memory\t", t)
print("\tmemory_reserved\t", r)
print("\tmemory_allocated\t", a)
print("\tfree inside reserved\t", f, "\n\n")
</code></pre>
<p>Output:</p>
<blockquote>
<p>Before Importing code_2.py<br>
total_memory	 25396969472<br>
memory_reserved	 0<br>
memory_allocated	 0<br>
free inside reserved	 0</p>
<p>Code_2.py Global context is executed!</p>
<p>After importing code_2.py<br>
total_memory	 25396969472<br>
memory_reserved	 0<br>
memory_allocated	 0<br>
free inside reserved	 0</p>
</blockquote>
<p>So the mystery still remains <img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=12" title=":thinking:" class="emoji" alt=":thinking:" loading="lazy" width="20" height="20"></p>
Out of memory when resuming DDP training

zzzf: <p>I got OOM when resuming DDP training:</p>
<pre><code class="lang-auto">device = torch.device("cuda:{}".format(local_rank))
model.to(device)
model.load_state_dict(torch.load(path, map_location=device))
DistributedDataParallel(model, device_ids=[local_rank])

...
optimizer.step() &lt;-- OOM here
...
</code></pre>
<p>After I change the <code>map_location</code> to <code>'cpu'</code>, the problem is fixed. But I’m wondering why the first case would fail?</p>
<pre><code class="lang-auto">device = torch.device("cuda:{}".format(local_rank))
model.load_state_dict(torch.load(path, map_location=device))
model.to(device)
DistributedDataParallel(model, device_ids=[local_rank])
</code></pre>
Insufficient GPU memory due to differences in memory size between the training machine and the deployed machine

Skyworker: <p>Because the GPU memory used for forward propagation during training is very large, but there is no such high GPU memory during actual deployment. Is there any way to reduce the video memory usage by an order of magnitude during deployment?<br>
Batch size is already 1.<br>
The main reason for the large memory usage is because the data is three-dimensional.</p>

ptrblck: <p>Since you won’t be calculating the gradients during deployment you should wrap the forward pass into a <code>with torch.no_grad()</code> or <code>with torch.inference_mode()</code> guard which won’t store any forward activations and will thus decrease the memory usage. The actual reduction in memory usage depends on the model architecture.</p>

Skyworker: <p>Using <code>with torch.no_grad()</code> or <code>with torch.inference_mode()</code> like this？</p>
<pre><code class="lang-auto">    def forward(self, x):
        with torch.no_grad():
            x1 = self.conv1_1(x)
            x1 = self.conv1_2(x1)
            x1 = self.conv1_3(x1)
</code></pre>
<p>even while training？directly add it in the model？</p>

ptrblck: <p>No, you shouldn’t use it during training as it will disable the gradient calculation as previously explained. Wrap the forward pass of the entire model in the guard during inference:</p>
<pre><code class="lang-python"># inference
model.eval()
with torch.no_grad():
    out = model(x)
</code></pre>

Skyworker: <p>Very useful! Thank you</p>
About torch.fx.passes.graph_drawer

MartinZhang: <p>Hi, today I try to use the torch.fx.passes.graph_drawer to get a vision of graphmodule</p>
<p>like this:</p>
<pre><code class="lang-auto">model = torch.load('graph_model_resnet50.pt')  # this is a graphModule from torch.fx.symbolic_trace

from torch.fx import passe

passes.graph_drawer(model, 'renset50')
</code></pre>
<p>but there will a error about:</p>
<pre><code class="lang-auto">TypeError: 'module' object is not callable
</code></pre>
<p>I am not find the reason about thie error.</p>

ptrblck: <p><code>torch.fx.passes.graph_drawer</code> is a module, not a method.<br>
Use <code>passes.graph_drawer.FxGraphDrawer(model, 'resnet50')</code> and it should work.</p>

MartinZhang: <p>thanks， <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>I found the FxGraphDrawer from graph_drawer.py<br>
ps: <a href="https://github.com/pytorch/pytorch/blob/master/torch/fx/passes/graph_drawer.py" rel="noopener nofollow ugc">pytorch/graph_drawer.py at master · pytorch/pytorch · GitHub</a></p>
<p>But when I use this basic usage</p>
<p>there not <code> get_dot_graph().create_svg()</code> api from FxGraphDrawer</p>
<pre><code class="lang-auto">Visualize a torch.fx.Graph with graphviz
Basic usage:
    g = FxGraphDrawer(symbolic_traced, "resnet18")
    with open("a.svg", "w") as f:
        f.write(g.get_dot_graph().create_svg())
</code></pre>
<p>and I try:</p>
<pre><code class="lang-auto">    g = FxGraphDrawer(symbolic_traced, "resnet18")
    with open("a.svg", "w") as f:
        f.write(g.get_main_dot_graph().__str__())
</code></pre>
<p>but the svg file it can not load.</p>
<p>So, how to do it , and success to save a svg file?</p>

MartinZhang: <p>Perhaps is the version of the problem <img src="https://discuss.pytorch.org/images/emoji/apple/face_with_head_bandage.png?v=12" title=":face_with_head_bandage:" class="emoji" alt=":face_with_head_bandage:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>This code works for me and creates an <code>svg</code> file:</p>
<pre><code class="lang-python">import torch
import torchvision.models as models
from torch.fx import passes, symbolic_trace

model = models.resnet18()
model = symbolic_trace(model)

g = passes.graph_drawer.FxGraphDrawer(model, 'resnet50')
with open("a.svg", "wb") as f:
    f.write(g.get_dot_graph().create_svg())
</code></pre>

MartinZhang: <p>Oh, awsome. It can work, thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>.</p>

ptrblck: <p>Good to hear it’s working for you, too!</p>
CondaHTTPError when trying to install pytorch CUDA 11.6 on Windows

michaelmorgann: <p>I get the following Error Code when I try to install PyTorch in Anaconda, I’m not sure what to do. Please Help:</p>
<p>(base) PS C:\Users******&gt; conda install pytorch torchvision torchaudio cudatoolkit=11.6 -c pytorch -c conda-forge<br>
Collecting package metadata (current_repodata.json): failed</p>
<p>CondaHTTPError: HTTP 000 CONNECTION FAILED for url <a href="https://conda.anaconda.org/pytorch/win-64/current_repodata.json" rel="noopener nofollow ugc">https://conda.anaconda.org/pytorch/win-64/current_repodata.json</a><br>
Elapsed: -</p>
<p>An HTTP error occurred when trying to retrieve this URL.<br>
HTTP errors are often intermittent, and a simple retry will get you on your way.<br>
‘<a href="https://conda.anaconda.org/pytorch/win-64" class="inline-onebox" rel="noopener nofollow ugc">Win 64 :: Anaconda.org</a>’</p>

areeb_adnan: <p>I have this same problem but can’t find solution</p>

Danxfj: <p>I have this problem too, I’m trying to find someway solve this problem.</p>
Scheduler state_dict size

RupertP: <p>Hello,</p>
<p>I am currently debugging an error which causes my program to crash when resuming a training. Resuming the scheduler causes an out of memory error and I might create a seperate thread about this problem.</p>
<p>Currently I am investigating my scheduler and I am wondering why the size of the saved <code>scheduler.state_dict()</code> is so big.</p>
<p>I use an SGD optimizer and the OneCycleLR scheduler.</p>
<pre><code class="lang-auto">optimizer = torch.optim.SGD(objective_params, lr=lr, momentum=mo, weight_decay=wd)
scheduler = OneCycleLR(optimizer, max_lr=lr, total_steps=conf.max_iter)
</code></pre>
<p>Now I immediatly save the state_dicts of the model, the optimizer and the scheduler.</p>
<pre><code>torch.save(model.state_dict(), modelpath)
torch.save(scheduler.state_dict(), schedpath")
torch.save(optimizer.state_dict(), optpath)
</code></pre>
<p>The files have the following sizes:</p>
<p>model:       852,7 MiB<br>
optimizer:  1,9 KiB<br>
scheduler: 851,9 MiB</p>
<p>Then I start the training and because I use mixed precision it takes some stepts until the optimizer actually performs the<code> step()</code> function.</p>
<p>After it has the optimizer has stepped the sizes of the saved state dicts is the following:</p>
<p>model: 852,7 MiB<br>
optimizer: 843,9 MiB<br>
scheduler: 1,7 GiB</p>
<p>Why the size of the scheduler so enourmous? It makes sense that SGD uses the same amount of memory as the model because every weight has a momentum value but I do not understand why the scheduler takes up so much memory.  Printing out the scheduler state_dict does not answer my question as it is basically empty:</p>
<p><code>{'total_steps': 200000, 'step_size_up': 59999.0, 'step_size_down': 140000.0, 'anneal_func': &lt;bound method OneCycleLR._annealing_cos of &lt;lib.lr.OneCycleLR object at 0x7fd8a1061390&gt;&gt;, 'cycle_momentum': True, 'use_beta1': False, 'base_lrs': [4e-05], 'last_epoch': 10, '_step_count': 10}</code></p>
<p>Greetings Rupert</p>

LutingWang: <p>Two years late…But I came across the same problem recently. The root cause is the <code>anneal_func</code> entry in the scheduler <code>state_dict</code>. Since <code>scheduler.anneal_func</code> is a reference to <code>OneCycleLR._annealing_cos</code>, the whole scheduler is saved along with the <code>scheduler.optimizer</code> attribute, which is normally not saved in <code>scheduler.state_dict</code>. The problem is, <code>scheduler.optimizer.param_groups</code> contains all the model parameters. That is why <code>scheduler.state_dict</code> takes up so much space.</p>
<p>For more info please refer to <a href="https://github.com/pytorch/pytorch/issues/42376" class="inline-onebox" rel="noopener nofollow ugc">torch.save() serializes a bound method in OneCycleLR's state_dict, causing CUDA problems · Issue #42376 · pytorch/pytorch · GitHub</a>.</p>
Torch.isnan/torch.isinf cost long in the first iteration step

ZiQing_Li: <p>The bug part resulting the long running time is listed here. <a href="https://github.com/saidwivedi/DSR/blob/7d55d173192a44ec567864befd2fb20e993098eb/dsr/losses/losses.py#L245-L282" rel="noopener nofollow ugc">Bug part</a></p>
<p>The code calculates loss for individual sample in batch to avoid single sample loss nan or inf.</p>
<p>The shapes for <code>loss_dsr_c</code> and <code>loss_dsr_mc</code> are same, which is [batch_size].</p>
<p>I tried to profile the code as shown below. <a href="https://github.com/saidwivedi/DSR/blob/7d55d173192a44ec567864befd2fb20e993098eb/dsr/losses/losses.py#L272-L282" rel="noopener nofollow ugc">Profiling part</a></p>
<pre><code class="lang-auto">start = time.time()
if torch.isnan(loss_dsr_c[idx]) or torch.isnan(loss_dsr_mc[idx]) or \
   torch.isinf(loss_dsr_c[idx]) or torch.isinf(loss_dsr_mc[idx]):
    pass
    # imgs, imgname, grphs = gt_batch['img'], gt_batch['imgname'], gt_batch['grph']
    # debug_rend_out(imgs, grphs, cur_rend_out, cur_dsr_mc_img_label, \
    #                cur_dsr_mc_dist_mat, idx)
    # logger.warning(f'loss is nan for {imgname[idx]}')
    # logger.warning(f'current_rend - {torch.unique(cur_rend_out)}')
    # logger.warning(f'Rend_DSR_C - {torch.unique(rend_dsr_c)}')
    # logger.warning(f'Rend_DSR_MC - {torch.unique(rend_dsr_mc)}')
    # loss_dsr_c[idx] = 0.
    # loss_dsr_mc[idx] = 0.
logger.info(f"Step {idx}: " + str(time.time()-start) + "s")
</code></pre>
<p>Here is the corresponding console output.</p>
<pre><code class="lang-auto">2022-09-09 14:24:05.891 | INFO     | dsr.losses.losses:sr_losses:467 - Step 1: 1.0544300079345703s
2022-09-09 14:24:05.892 | INFO     | dsr.losses.losses:sr_losses:467 - Step 2: 6.937980651855469e-05s
2022-09-09 14:24:05.892 | INFO     | dsr.losses.losses:sr_losses:467 - Step 3: 6.127357482910156e-05s
2022-09-09 14:24:05.893 | INFO     | dsr.losses.losses:sr_losses:467 - Step 4: 0.00018215179443359375s
2022-09-09 14:24:05.893 | INFO     | dsr.losses.losses:sr_losses:467 - Step 5: 5.7697296142578125e-05s
2022-09-09 14:24:05.893 | INFO     | dsr.losses.losses:sr_losses:467 - Step 6: 7.033348083496094e-05s
2022-09-09 14:24:05.894 | INFO     | dsr.losses.losses:sr_losses:467 - Step 7: 7.462501525878906e-05s
2022-09-09 14:24:05.894 | INFO     | dsr.losses.losses:sr_losses:467 - Step 8: 7.319450378417969e-05s
..................................................
</code></pre>
<p>It is weird to find that only in the first iteration step, the cost time is much longer than the others while running the same code (i.e. torch.isnan and torch.isinf).</p>
Bug[wnnpack-cpp-51]-could-not-initialize-nnpack-reason-unsupported-hardware

Meer_Hamza: <p>This error occurs while I run torch on my flask server, can anyone tell me how to solve this problem I am using digital ocean VPS</p>

ptrblck: <p>I guess NNPack depends on some AVX instructions which might not be available on your CPU so you might need to build PyTorch without NNPack.</p>

Meer_Hamza: <p>How to do that because all my packages are installed in a virtualenv.</p>

Meer_Hamza: <p><code>sudo apt update &amp;&amp; sudo apt upgrade</code><br>
<code>sudo apt install libopenblas-dev libblas-dev m4 cmake cython python3-dev python3-yaml python3-setuptools</code><br>
<code>cd ~</code><br>
<code>mkdir pytorch_install &amp;&amp; cd pytorch_install</code><br>
<code>git clone --recursive https://github.com/pytorch/pytorch</code><br>
<code>cd pytorch</code></p>
<p><code>export MAX_JOBS=2</code><br>
<code>export NO_CUDA=1</code><br>
<code>export NO_DISTRIBUTED=1</code><br>
<code>export NO_MKLDNN=1</code><br>
<code>export USE_NNPACK=0</code><br>
<code>export USE_QNNPACK=1</code><br>
<code>export USE_CFLAGS=1</code><br>
<code>export CFLAGS=’-mfpu=neon-vfpv4 -mtune=cortex-a17 -march=armv7ve -mfloat-abi=hard’</code></p>
<p><code>python3 setup.py build # clean first if needed</code><br>
<code>sudo -E python3 setup.py install</code></p>
<p>I have tried this method to install the torch but I got this error:</p>
<p><strong>Building wheel torch-1.13.0a0+git942c0f3</strong><br>
<strong>– Building version 1.13.0a0+git942c0f3</strong><br>
<strong>cmake --build . --target install --config Release – -j 1</strong><br>
<strong>gmake: Makefile: No such file or directory</strong><br>
<strong>gmake: *** No rule to make target ‘Makefile’.  Stop.</strong></p>

Meer_Hamza: <p>from typing_extensions import Literal<br>
ModuleNotFoundError: No module named ‘typing_extensions’<br>
CMake Error at cmake/Codegen.cmake:197 (message):<br>
Failed to get generated_headers list<br>
Call Stack (most recent call first):<br>
caffe2/CMakeLists.txt:2 (include)</p>
<p><strong>Now its giving me the typing_extensions not found</strong></p>

ptrblck: <p>You can either install all dependencies manually via <code>pip install ...</code> or just install them via <code>pip install -r requirements.txt</code></p>

Meer_Hamza: <p>pytorch_install/pytorch$ pip3 install typing_extensions<br>
Requirement already satisfied: typing_extensions in /home/hamza/xtremeremoval/env/lib/python3.10/site-packages (4.3.0)</p>
<p><strong>But requirement already satisfied</strong></p>

Meer_Hamza: <p>I found one solution to this problem</p>
<p>I have installed cmake on Ubuntu Linux now the server is working fine, the unsupported error has been gone</p>
<p><strong>sudo apt update</strong><br>
<strong>sudo apt cmake</strong><br>
check the library:<br>
<strong>cmake --version</strong></p>
Missing filter for ROCm for downloading wheels

kunaltyagi: <p><a href="https://download.pytorch.org/whl/" rel="noopener nofollow ugc">https://download.pytorch.org/whl/</a> is quite useful for automation like CI. Another feature that I really like is the platform filter, example:</p>
<ul>
<li>
<code>https://download.pytorch.org/whl/${arch}</code>, arch = cpu, cu116, etc.</li>
</ul>
<p>However, I noticed that there is no such filter for ROCm. None of the following work, and trying these gives unauthorized access XML as return.</p>
<ul>
<li><a href="https://download.pytorch.org/whl//rocm/" rel="noopener nofollow ugc">https://download.pytorch.org/whl//rocm/</a></li>
<li>prefix/rocm511/ (using prefix since new users are limited at 2 links only. prefix = <code>https://download.pytorch.org/whl/</code>)</li>
<li>prefix/rocm5.1.1/</li>
</ul>
<p>Any idea what I can use for filtering wheels for ROCm?</p>
In DistributedDataParallel training additional n-1 processes are using memory on the first GPU, n being the number of GPUs

Ratul_Ghosh: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/c/4c6923fbc5118226c41964f3200945a788ac861d.png" data-download-href="https://discuss.pytorch.org/uploads/default/4c6923fbc5118226c41964f3200945a788ac861d" title="Screen Shot 2022-09-08 at 5.46.25 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/c/4c6923fbc5118226c41964f3200945a788ac861d_2_308x500.png" alt="Screen Shot 2022-09-08 at 5.46.25 PM" data-base62-sha1="aTXy7BCTpvCTuVsRPtrX2gL0hop" width="308" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/c/4c6923fbc5118226c41964f3200945a788ac861d_2_308x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/c/4c6923fbc5118226c41964f3200945a788ac861d_2_462x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/c/4c6923fbc5118226c41964f3200945a788ac861d_2_616x1000.png 2x" data-dominant-color="393A34"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2022-09-08 at 5.46.25 PM</span><span class="informations">778×1260 204 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
How get indices of elements in an array on another array?

Tony_Gracious: <pre><code class="lang-auto">a = torch.randint(0,5,(10,))
b =  torch.tensor([3,4,1,2, 0] )
</code></pre>
<p>I want to find all positions of elements in b in array a. Currently, I use this</p>
<p><code>[a.eq(b[i]).nonzero(as_tuple=True) for i in range(b.shape[0])]</code>.</p>
<p>Is there any better way to do this?</p>

Matias_Vasquez: <p>You can do something like this.</p>
<pre><code class="lang-python">((a - b.unsqueeze(0).T) == 0).nonzero()
</code></pre>
<p>The output format is different, but the idea is the same.</p>
<h3>
<a name="example-1" class="anchor" href="#example-1"></a>Example</h3>
<p>Where</p>
<pre><code class="lang-python">#a = tensor([3, 3, 3, 2, 4, 2, 1, 3, 4, 0])
#b = tensor([3, 4, 1, 2, 0])
</code></pre>
<ul>
<li>Your format</li>
</ul>
<pre><code class="lang-python"># Output: 
[(tensor([0, 1, 2, 7]),), (tensor([4, 8]),), (tensor([6]),), (tensor([3, 5]),), (tensor([9]),)]
</code></pre>
<ul>
<li>New format</li>
</ul>
<pre><code class="lang-python"># Output:
tensor([[0, 0],
        [0, 1],
        [0, 2],
        [0, 7],
        [1, 4],
        [1, 8],
        [2, 6],
        [3, 3],
        [3, 5],
        [4, 9]])
</code></pre>
<p>Where the first column corresponds to the index for <code>b</code> and the second column for <code>a</code>.</p>

Tony_Gracious: <pre><code class="lang-auto"># Output:
tensor([[0, 0],
        [0, 1],
        [0, 2],
        [0, 7],
        [1, 4],
        [1, 8],
        [2, 6],
        [3, 3],
        [3, 5],
        [4, 9]])
</code></pre>
<p>Is there way to do a mean-pooling with respect index for b.  For example, I want to do a grouby mean like this,</p>
<pre><code class="lang-auto"> [t[t[:,0].float()==i][:, 1].float().mean() for i in range(4)]
</code></pre>

Tony_Gracious: <p>I got it, you can do the following,</p>
<pre><code class="lang-auto"> index = ((a - b.unsqueeze(0).T)== 0 ).float()
r  = index @ a.unsqueeze(dim=1 )/ (index.sum(dim=1, keepdim=True)
</code></pre>

KMakowsky: <p>Unfortunately this approach will use immense memory since (a - b.unsqueeze(0).T) spans a whole matrix over a and b. It was not suitable for my purpose (a and b had more than 150000 elements, this results in 22,5 billion elements in memory ~ 21GB).</p>
<p>My solution is:</p>
<pre><code class="lang-auto">def index_of_a_in_b(a, b):
  b_indices = torch.where(torch.isin(b, a))[0]
  b_values = b[b_indices]
  return b_indices[b_values.argsort()[a.argsort().argsort()]]
</code></pre>
Is there e a PyTorch implementation of MobileNet V1?

Mohamed_Ahmad: <p>Good day everyone,</p>
<p>As the question states, I’m wondering if there exists an offical model representation of Mobilenet version 1? I know that version 2 exists here <a href="https://github.com/pytorch/vision/blob/master/torchvision/models/mobilenet.py" rel="nofollow noopener">https://github.com/pytorch/vision/blob/master/torchvision/models/mobilenet.py</a> but is there a simillar representation for version1?</p>
<p>Thanks a lot for the help!!</p>

littlebee: <p>Also interested in the answer to this.   MobileNet V1 is a rockstar performer on TFLite and Raspberry Pi 4.</p>
<p>Referring specifically to the quantized version for tflite available here:      <a href="https://raw.githubusercontent.com/google-coral/test_data/master/ssd_mobilenet_v1_coco_quant_postprocess.tflite" rel="noopener nofollow ugc">https://raw.githubusercontent.com/google-coral/test_data/master/ssd_mobilenet_v1_coco_quant_postprocess.tflite</a></p>
Modifying graph convolution layer (message passing)

emad1: <p>Hi, I am new to pytorch geometrics. I want to make a change to a graph convolution layer message passing scheme. more specifically I want to add some equation to the massage passing so I can remove some nodes from aggregation. I have been trying to figure out that what part of this code I can modify to achieve this .the following is the source code <a href="https://pytorch-geometric.readthedocs.io/en/latest/_modules/torch_geometric/nn/conv/graph_conv.html#GraphConv" rel="noopener nofollow ugc">TORCH_GEOMETRIC.NN.CONV.GRAPH_CONV</a> .</p>
Combine Use of Multi-GPU and Multi-CPU

Fhunmie: <p>I have access to HPC node, The maximum wall time for the GPU node I have access to is 12 hours. I have access to a maximum of 4 Tesla V100-PCIE-16GB. And about 36 CPU cores. I wondered if DDP could allow me to use all the cores alongside GPUs. I would love a link to a tutorial resource that is possible. Thanks.</p>

wi-sh: <p>Probably looking something similar to this <a href="https://researchcomputing.princeton.edu/support/knowledge-base/pytorch" rel="noopener nofollow ugc">https://researchcomputing.princeton.edu/support/knowledge-base/pytorch</a></p>
[Profiler] What are the device/host time definitions, how to interpret them

SJJK: <p>Hi, please bear with this simple question. When using the Pytorch profiler, I found the device/host time confusing. Could anyone please share some guidance on their specific definition?</p>
<p>For example for this output, they vary quite a lot. How may I interpret the chart to better understand the performance bottleneck?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/7/9763a81c3d09f1cbd60ff00d41441c3000c03ea6.png" data-download-href="https://discuss.pytorch.org/uploads/default/9763a81c3d09f1cbd60ff00d41441c3000c03ea6" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/7/9763a81c3d09f1cbd60ff00d41441c3000c03ea6_2_690x325.png" alt="image" data-base62-sha1="lBfBUzDBbyxoAGZyZeWHZHDwM8C" width="690" height="325" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/7/9763a81c3d09f1cbd60ff00d41441c3000c03ea6_2_690x325.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/7/9763a81c3d09f1cbd60ff00d41441c3000c03ea6_2_1035x487.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/7/9763a81c3d09f1cbd60ff00d41441c3000c03ea6_2_1380x650.png 2x" data-dominant-color="E8E4E6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1448×683 109 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Any help would be appreciated.</p>
Verbose Warning with torch.matrix_exp

okfallah: <p>On torch <code>1.12.1+cu113</code> have a tensor <code>T</code> with <code>T.shape = torch.Size([64, 512, 512])</code> on a cuda device. When I call <code>torch.matrix_exp(T)</code> I get the following warning:</p>
<pre><code class="lang-auto">&lt;string&gt;:1: UserWarning: An output with one or more elements was resized since it had shape [45, 512, 512], 
which does not match the required output shape [1, 45, 512, 512]. This behavior is deprecated, 
and in a future PyTorch release outputs will not be resized unless they have zero elements. 
You can explicitly reuse an out tensor t by resizing it, inplace, to zero elements with 
t.resize_(0). (Triggered internally at  ../aten/src/ATen/native/Resize.cpp:17.)
</code></pre>
<p>Note that I <em>cannot</em> reproduce it with this simple case:</p>
<pre><code class="lang-auto">T = torch.randn(64, 512, 512, device="cuda:0")
print(T.shape)
print(torch.matrix_exp(T).shape)
</code></pre>
<p>I believe this has to do with how matrix_exp handles different order approximations (hence the arbitrary 45 shape in the warning message). Any advice on how to fix this? Or at least suppress this warning message (it is coming multiple times an iteration for me).</p>
How to pad with a (CNN) custom value

M2M: <p>Hi everyone,</p>
<p>This could be a silly question. I want to know how to pad my 3dCNN with a given (custom) number, not just zeros.</p>
<p>Also, if someone can refer me to a resource which gives a comprehensive explanation of padding modes would be very helpful.</p>
<p><strong>padding_mode</strong> (<em>string</em> <em>,</em> <em>optional</em> ) – <code>'zeros'</code> , <code>'reflect'</code> , <code>'replicate'</code> or <code>'circular'</code></p>

azhanmohammed: <p>To pad a given with a custom number, you will have to use the functionality <code>torch.nn.functional.pad()</code> mentioned in the docs <a href="https://pytorch.org/docs/stable/generated/torch.nn.functional.pad.html" rel="noopener nofollow ugc">here</a>.<br>
A simple implementation can be:</p>
<pre><code class="lang-python">class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        # pads with custom float value and then performs 3d convolution
        # no padding at convolution step
        self.conv1 = nn.Conv3d(3, 16, 3, stride=2, padding=0)
        self.p3d = (1, 1, 1, 1, 1, 1) # pad by (1, 1), (1, 1), and (1, 1)
        
    def forward(self, x):
        #p3d is dimensions to pad, constant means constant value padding, and 0 is the number you want to pad with
        x = torch.nn.functional.pad(x, self.p3d, "constant", 0)
        x = self.conv1(x)
        return x
</code></pre>
<p>This way you can pad with any custom number you want.</p>

M2M: <p>Thanks a lot <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>It is interesting how things are done in PyTorch</p>
Dataset in shared memory

rvandeghen: <p>Hi,<br>
Imagine that you have enough RAM to load and store all the <code>images</code> and the <code>targets</code> of your dataset in a <code>list</code> during the <code>__init__</code> of your <code>Dataset</code> class. When training a model using <code>torchrun</code>, each gpu will create its own <code>Dataset</code>, thus leading to <code>N*dataset_size</code> of RAM used.<br>
Is it possible to share the <code>images</code> and <code>targets</code> lists to all processes without replications, so that the memory consumption is still <code>dataset_size</code> ?</p>
<p>FYI, my <code>Dataset</code> inherits from <a href="https://github.com/pytorch/vision/blob/main/torchvision/datasets/coco.py" rel="noopener nofollow ugc">COCO</a>.</p>
GPT-2 in DNA data

mdelas: <p>Dear community,</p>
<p>I’m trying to build a GPT-2 transformer from scratch (without any pre-train model) with DNA sequences in order to generate DNA sequences on top of smaller ones. I am a bit stuck and I couldn’t find any repo applying this kind of decoder-transformer with a DNA background, to have some clues in what’s the best tokenization, and some other technical choices…</p>
<p>Does someone have any references or think that’s a good idea?</p>
<p>Thank you in advance!</p>
Should many intermediate computations be left in the forward propagation of the neural network?

Nguyen_Duy: <p>Hello everyone,<br>
I have a question as follows:<br>
I need to use a temporary variable during the transition of an artificial neural network. Specifically, assuming the forward propagation of that neural network has an input of x, I use a variable y transformed from x over 3-4 operations in the forward function. Should I do that? Or should I create the variable y right from the time of sample generation.</p>
<pre><code class="lang-auto">def forward(self, x, mask):
  z = self.linear1(x)
  # Some computations create y from x (examples)
  total = torch.sum(x, dim=(1, 2))
  total = total * mask
  y = x / total
  # Continue forward propagation
  result = z + self.linear2(y)
  return result
</code></pre>
<p>or</p>
<pre><code class="lang-auto"># Create y from sampling 
def forward(self, x, mask, y):
  z = self.linear1(x)
  result = z + self.linear2(y)
  return result
</code></pre>
<p>I want to ask is there any difference between the above two ways about:</p>
<ul>
<li>GPU memory used when training</li>
<li>Size of the model<br>
Sorry if my expression is not good. Thanks for any help</li>
</ul>
When training a model, which is more important, minimizing training loss or maximizing test accuracy?

BobKim: <p>Hello!</p>
<p>While training the model, I have some question about the training loss.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/4/e/4e8c1715ae759bef7e552cc053a0993333f5e1e5.png" alt="QA_CIFAR10_Train" data-base62-sha1="bcRnQ3ssJa4Na5xq1toWfEagsDz" width="320" height="240"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/2/d/2d75fff81c15ef45f5e009b3c6d08edec93ce739.png" alt="QA_CIFAR10_Test" data-base62-sha1="6uaiAdVXO2zYbMBUaoIKTsiyWcp" width="320" height="240"></p>
<p>This plot is the result of training with different A and B optimizers.<br>
The difference between the training loss values of A and B is large, but the test accuracy is almost the same, or A has higher accuracy.<br>
If so, does the training loss have no significant relationship with the test accuracy?<br>
In my opinion, from a traditional optimizer perspective, an optimizer that minimizes training loss is good.<br>
What do you think about this issue?</p>
Efficient net as backbone feature extractor

treadstone: <p>How to use efficientNet as backbone CNN model for feature extraction, so that embeddings of images can be generated.Most of example on GitHub use 4 layer ConvNet so I can not understand how to use same thing for large CNN model. Like there are implementation of efficient-net for Torch, so what steps I need to use them as feature extractor?<br>
I am using this efficient net code which implemented network in PyTorch -</p><aside class="onebox githubblob">
  <header class="source">

      <a href="https://github.com/lukemelas/EfficientNet-PyTorch/blob/master/efficientnet_pytorch/model.py" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/lukemelas/EfficientNet-PyTorch/blob/master/efficientnet_pytorch/model.py" target="_blank" rel="noopener nofollow ugc">lukemelas/EfficientNet-PyTorch/blob/master/efficientnet_pytorch/model.py</a></h4>


    <pre><code class="lang-py">"""model.py - Model and module class for EfficientNet.
   They are built to mirror those in the official TensorFlow implementation.
"""

# Author: lukemelas (github username)
# Github repo: https://github.com/lukemelas/EfficientNet-PyTorch
# With adjustments and added comments by workingcoder (github username).

import torch
from torch import nn
from torch.nn import functional as F
from .utils import (
    round_filters,
    round_repeats,
    drop_connect,
    get_same_padding_conv2d,
    get_model_params,
    efficientnet_params,
    load_pretrained_weights,
    Swish,
</code></pre>


  This file has been truncated. <a href="https://github.com/lukemelas/EfficientNet-PyTorch/blob/master/efficientnet_pytorch/model.py" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


darshvirbelandis: <p>Did you figure this out? Currently trying to implement this as a feature extractor and was a bit confused on which layer to access and how to access the specific layer for feature output.</p>
Function to compute intersection over union of boxes

Mika_S: <p>Is there code in python available for intersection over union (iou) of bounding boxes?</p>
<p>I have been able to find non maximal suppression function in tensorflow which must be computing iou but I have not found a public function.</p>

amdegroot: <p>It’s not included in the PyTorch tensor library, but I compute iou (jaccard index) of bounding boxes using PyTorch tensors here: <a href="https://github.com/amdegroot/ssd.pytorch/blob/master/box_utils.py#L33-L70" rel="nofollow noopener">https://github.com/amdegroot/ssd.pytorch/blob/master/box_utils.py#L33-L70</a>.</p>

Vijay_Dubey: <p><a class="mention" href="/u/amdegroot">@amdegroot</a><br>
This link does not open.<br>
Can you please help.<br>
I need to see the function on how to compute iou of bounding boxes</p>

amdegroot: <p>Sorry for just now seeing this, but to update the link: <a href="https://github.com/amdegroot/ssd.pytorch/blob/master/layers/box_utils.py#L48" rel="nofollow noopener">https://github.com/amdegroot/ssd.pytorch/blob/master/layers/box_utils.py#L48</a>.</p>

nscozzaro: <p>Torchvision has a function to compute iou of boxes:<br>
<a href="https://pytorch.org/vision/main/ops.html#box-operators" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/vision/main/ops.html#box-operators</a></p>
Floating point precision with torch operations

lkdci: <p>The theoretical floating point precision is 2**32, which is ~4.3e9. I noted a weird behavior with <code>torch.eq</code> operation, seems that the limit is around at 2.9e-8, see the example bellow.<br>
Is there any documentation about those values? which epsilon values are recommended to use to avoid numerical errors when using fp32 or fp16?</p>
<pre><code class="lang-auto">torch.tensor([1.]).eq(1.)
Out[35]: tensor([True])
torch.tensor([1. - 3e-8]).eq(1.)
Out[36]: tensor([False])
torch.tensor([1. - 2.9e-8]).eq(1.)
Out[37]: tensor([True])
</code></pre>

KFrank: <p>Hi lkdci!</p>
<aside class="quote no-group" data-username="lkdci" data-post="1" data-topic="160941" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/7993a0/48.png" class="avatar"> lkdci:</div>
<blockquote>
<p>The theoretical floating point precision is 2**32</p>
</blockquote>
</aside>
<p>Your misconception here is that the full 32 bits of a 32-bit single-precision<br>
floating-point number are used for precision.  But 8 of those bits are used<br>
for the number’s (binary) exponent, which is what gives floating-point<br>
numbers their large dynamic range.</p>
<p>That leaves you then with (roughly speaking) only 24 bits of precision,<br>
which translates, as you have noted, to about eight decimal digits of<br>
precision.</p>
<p>Wikipedia’s <a href="https://en.wikipedia.org/wiki/Single-precision_floating-point_format" rel="noopener nofollow ugc">single-precision-floating-point entry</a> gives a good overview<br>
of this.</p>
<p>(Beware, also, of pytorch’s dreaded <a href="https://pytorch.org/docs/1.12/notes/cuda.html#tensorfloat-32-tf32-on-ampere-devices" rel="noopener nofollow ugc">TensorFloat-32</a> that automatically<br>
and silently gives you <em>distinctly less</em> precision for some tensor<br>
operations on some recent fancy gpus.)</p>
<p>Best.</p>
<p>K. Frank</p>
Predefined model and repeated training calls using Distributed Data Parallel

AbinandN: <p>Hello,</p>
<p>I am trying to implement DDP to speed training in my project. I attempted to follow multiple tutorials (<a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" rel="noopener nofollow ugc">example</a>). In all these tutorials, the model and dataloaders are created inside the spawned processes. This does not work for me, as I have to produce some metrics for the model and modify it between epochs and repeat this process numerous times.  Is it possible to create the model in the main process, where I can perform the inter-epoch modifications and have the workers run the training loop again upon command from the main process?</p>
<p>For those curious, I am trying to use DDP with the Hardware Aware Quantization framework.</p>

H-Huang: <aside class="quote no-group" data-username="AbinandN" data-post="1" data-topic="160804">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/abinandn/48/53004_2.png" class="avatar"> AbinandN:</div>
<blockquote>
<p>I have to produce some metrics for the model and modify it between epochs and repeat this process numerous times. Is it possible to create the model in the main process, where I can perform the inter-epoch modifications and have the workers run the training loop again upon command from the main process?</p>
</blockquote>
</aside>
<p>If I understand correctly, you are trying to modify the model architecture between epochs during training. This doesn’t fit well into the DDP paradigm as it expects the model to be replicated across all workers so somehow this needs to be synced correctly.</p>
<p>I think <a href="https://pytorch.org/docs/master/rpc.html?highlight=rpc" rel="noopener nofollow ugc">RPC</a> could be a potential solution for you. You can store the model on the main process (rank 0), perform updates, and use RPC calls passing in the model.state_dict to have workers execute the training loop.</p>

kumpera: <p>As an alternative, you can rewrap your model with DDP again after modifying it.</p>

AbinandN: <p>Thanks.</p>
<p>I’ll look into this.</p>

AbinandN: <p>That would be an ideal solution, but the modifications will happen on the master process. How do I communicate the updated model to the other processes?</p>
What is tracing?

moth: <p>I have read the documentation provided in <a href="https://pytorch.org/docs/stable/generated/torch.jit.trace.html#torch-jit-trace" rel="noopener nofollow ugc">torch.jit.trace</a>, still the concept does not come clear to me.</p>
<p>What is the purpose of using <code>torch.jit.trace</code>? What are the advantages of tracing a model?</p>
Torch.empty() does not allocate memory?

iamhatesz: <p>I found out that calling <code>torch.empty()</code> does not allocate memory to fit future data. Is it an intended behavior? I believed that <code>torch.empty()</code> works like any other construction methods, but without filling in any values.</p>
<p><a href="https://pytorch.org/docs/stable/torch.html#torch.empty" rel="noopener nofollow ugc">Docs</a> says:</p>
<blockquote>
<p>Returns a tensor filled with uninitialized data.</p>
</blockquote>
<p>Which is ambiguous to me.</p>

ptrblck: <p><code>torch.empty</code> does allocate memory, but does not initialize this memory space.<br>
I.e. the values in your tensor will just be interpreted as the bits which are currently in the allocated memory space.</p>

iamhatesz: <p>Are you sure?</p>
<p>Here’s the output from the memory profiler:</p>
<pre><code class="lang-auto">Line #    Mem usage    Increment   Line Contents
================================================
     5     53.1 MiB     53.1 MiB   @profile
     6                             def main():
     7     53.2 MiB      0.1 MiB       t = torch.empty((1000000, 1000), dtype=torch.float, device='cpu')
     8     53.2 MiB      0.0 MiB       del t
     9   3868.3 MiB   3815.1 MiB       t = torch.zeros((1000000, 1000), dtype=torch.float, device='cpu')
    10     53.6 MiB      0.0 MiB       del t
</code></pre>

ptrblck: <p>Yes, otherwise you couldn’t index the tensor or work in some other way.<br>
I’m not sure, how the memory profiler works, but using the <code>process_memory_info()</code> data you’ll get the expected result:</p>
<pre><code class="lang-python">import torch

import os
import psutil
process = psutil.Process(os.getpid())
mem_t0 = process.memory_info().data
print(mem_t0)

x = torch.empty(1000, 1000, dtype=torch.float32)

mem_t1 =  process.memory_info().data
mem_expected = x.numel() * 4 /1024 # in kBytes
print('Expected {} kBytes'.format(mem_expected))
print('Actual {} kBytes'.format((mem_t1 - mem_t0) / 1024))
&gt; Expected 3906.25 kBytes
&gt; Actual 3908.0 kBytes
</code></pre>

myuser: <p>This is strange. Using the following code, I encounter a “Out Of Memory” error after a specific number of iterations.</p>
<pre><code class="lang-python">a = torch.empty(b.shape)
for i, batch in enumerate(train_dl):
    print(i, flush=True)
    a[batch_size*i : batch_size*(i+1)] = model(batch).detach()
</code></pre>
<p>When I change <code>torch.empty</code> to <code>torch.zeros</code>, I get the OOM error immediately without a single iteration.</p>

ptrblck: <p>Based on your code snippet it seems you are initializing <code>a</code> on the CPU, so it’s unclear why a change from <code>torch.empty</code> to <code>torch.zeros</code> would cause an OOM on the GPU as both tensors will be allocated on the CPU by default or are you seeing a host OOM?</p>

myuser: <p>Here, I am not using a GPU. It’s a host OOM.</p>

ptrblck: <p>How much RAM does your system have and what does the error message tell you?</p>
Best model of neural network to fit specific dataset in regression task

jahanifar: <p>Hello,<br>
I have a training dataset like below:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/a/6/a6297be5aaf84ae657d9432f40b45736a5215fb9.png" alt="image" data-base62-sha1="nHW8CGLkUAzmmJ8ELswyscaepnj" width="496" height="371"></p>
<p>and I want to implement a NN model for regression to be able predict below data:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/1/9/192007313892cd75e872592ae47abe9452a3d3f8.png" alt="image" data-base62-sha1="3AgyvIobUzdi5ntYEndYq3YCog8" width="496" height="371"></p>
<p>Could you please advise what is the best way to impalement a neural network to perform the task successfully? for example how many hidden layer should I use and how many units should I consider in each layer and which activation function and dropout value should I consider. Thanks</p>
How to concatenate LibTorch tensors created with a multi-thread process std::thread in C++?

Alio: <p>A multi-thread process in C++ returns tensors and I want to concatenate them into one tensor in order.</p>
<p>In C++ I have a single function which returns a single 1x8 tensor.<br>
I call this function multiple times simultaneously with <code>std::thread</code> and I want to concatenate the tensors it returns into one large tensor. For example, I call it 12 times I expect to have a 12x8 tensor when it’s finished.</p>
<p>I’m aware I could just have the function return a 12x8 tensor, but I need to solve the problem of how to grab the tensors produced during the multithreading process.</p>
<p>In my attempt below I try to concatenate the tensors into the <code>all_episode_steps</code> tensor but this returns an error.<br>
If you comment out the <code>all_episode_steps</code> line and put <code>std::cout &lt;&lt; one;</code> in the <code>get_tensors</code> function above the return statement you see it appears to be using multithreading to create the tensors with no problems.</p>
<pre><code>#include &lt;torch/torch.h&gt;

torch::Tensor get_tensors(int id) {
    torch::Tensor one = torch::rand({8});
    return one.unsqueeze(0);
}

torch::Tensor all_episode_steps;
int main() {

    std::thread ths[100];

    for (int id=0; id&lt;12; id++) {

        ths[id] = std::thread(get_tensors, id);

        all_episode_steps = torch::cat({ths[id], all_episode_steps});
    }
    for (int id=0; id&lt;12; id++) {
        ths[id].join();
    }

}
</code></pre>

Andrew_Edmondson: <p>I believe it is because you’re not initializing all_episode_steps before trying to cat it so there is nothing to cat with whatever is returned from the threads.</p>
How is dim for torch.matmul determined?

skerlet_flandorle: <p>I know that the attention code uses mutuml like this and that this code works<br>
But using matmul under the exact same conditions gives an error Why is this?<br>
I want to control dim to account for batch and attention_haed, but how do I control the calculated dimension of matmul?<br>
attencode and print output</p>
<pre><code class="lang-auto">        attn = self.dropout(attn)

        print("attn",attn.size())
        print(value.size())
        context = torch.matmul(attn, value).transpose(1, 2)
        context = context.contiguous().view(batch_size, -1, self.d_model)


attn torch.Size([4, 16, 100, 100])
torch.Size([4, 16, 100, 32])
</code></pre>
<p>my test code and error</p>
<pre><code class="lang-auto">v=torch.randn(4, 16, 100, 100)
at=torch.randn(4, 16, 100,32)
print("aa")
context = torch.matmul(at, v)
pritn(context.size())
</code></pre>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "a5atten.py", line 129, in &lt;module&gt;
    context = torch.matmul(at, v)
RuntimeError: Expected size for first two dimensions of batch2 tensor to be: [64, 32] but got: [64, 100].
</code></pre>

tom: <p>The error message is a bit confusing here because it comes from <code>torch.bmm</code> that <code>matmul</code> reduces to (I think), but the problem is that the second to last dimension of the right argument (<code>v</code>) should match the last dimension of the left (<code>at</code>), as this is the contraction dimension.</p>
<p>Best regards</p>
<p>Thomas</p>

InnovArul: <p>Just adding to <a class="mention" href="/u/tom">@tom</a>’s answer: You could use <code>torch.einsum()</code> to write more readable code.</p>
<p><a href="https://pytorch.org/docs/stable/generated/torch.einsum.html" class="onebox" target="_blank" rel="noopener">https://pytorch.org/docs/stable/generated/torch.einsum.html</a></p>
<pre><code class="lang-auto"># attention operation
q = torch.randn(4, 16, 100, 32) # batch, head, N, dim
k = torch.randn(4, 16, 100, 32)
v = torch.randn(4, 16, 100, 32)

attention_logits = torch.einsum("bhnd,bhmd-&gt;bhnm", q, k)   # 4, 16, 100, 100
attention_logits = attention_logits.softmax(dim=-1)
context = torch.einsum("bhnm,bhmd-&gt;bhnd", attention_logits, v)
pritn(context.size())
</code></pre>