Torch Profiler: GPU substantially slower than CPU

mloewen: <p><strong>Synopsis:</strong> Training and inference on a GPU is <em>dramatically</em> slower than on any CPU.</p>
<p><strong>Setup:</strong></p>
<ul>
<li>Training a highly customized Transformer model on an Azure VM (Standard NC6s v3 [6 vcpus, 112 GiB memory]) with a Tesla V100 (Driver Version: 550.54.15 &amp; CUDA Version: 12.4).</li>
<li>The dataset is not very large (e.g. 1 GB) with dimensions [12000, 51, 48] using mini-batches of size 256.</li>
<li>All data are loaded into tensors and sent to GPU memory (e.g. <em>to(device)</em>) before even instantiating the model.</li>
<li>model is sent <em>to(device)</em> and then compiled via <em>model = torch.compile(model, mode=“reduce-overhead”)</em>.</li>
<li>Call model training via <em>torch.autocast(device_type=device, dtype=torch.bfloat16, enabled=False)</em>.</li>
</ul>
<p><strong>Situation:</strong> Using Torch’s built-in Profiler, I am noticing a dramatic slowdown when using GPU/CUDA rather than calculating everything on CPU. I do not believe this has anything to do with data flowing between the CPU and GPU given my comments above, but could be wrong.</p>
<p>The following two tables are exactly the same datasets and run on the same machine. The only difference between the CPU runtime and the GPU/CUDA runtime is where the data is being stored (e.g. they are both Torch.Tensors sent to(device)) and where compute occurs.</p>
<p><strong>What’s Weird:</strong> <em>model_inference</em> is 27.5x slower on GPU than on CPU <em>and</em> model_inference is called 3x times in the CUDA version…</p>
<hr>
<p><strong>CPU Runtime</strong></p>
<div class="md-table">
<table>
<thead>
<tr>
<th>Name</th>
<th>Self CPU %</th>
<th>Self CPU</th>
<th>CPU total %</th>
<th>CPU total</th>
<th>CPU time avg</th>
<th># of Calls</th>
</tr>
</thead>
<tbody>
<tr>
<td>model_inference</td>
<td>1.90%</td>
<td>496.389ms</td>
<td>100.00%</td>
<td><strong>26.178s</strong></td>
<td><strong>26.178s</strong></td>
<td>1</td>
</tr>
<tr>
<td>aten::empty</td>
<td>0.55%</td>
<td>144.920ms</td>
<td>0.55%</td>
<td>144.920ms</td>
<td>19.006us</td>
<td>7625</td>
</tr>
<tr>
<td>aten::random_</td>
<td>0.00%</td>
<td>12.914us</td>
<td>0.00%</td>
<td>12.914us</td>
<td>12.914us</td>
<td>1</td>
</tr>
<tr>
<td>aten::item</td>
<td>0.00%</td>
<td>3.033us</td>
<td>0.00%</td>
<td>4.424us</td>
<td>4.424us</td>
<td>1</td>
</tr>
<tr>
<td>aten::_local_scalar_dense</td>
<td>0.00%</td>
<td>1.391us</td>
<td>0.00%</td>
<td>1.391us</td>
<td>1.391us</td>
<td>1</td>
</tr>
<tr>
<td>enumerate(DataLoader)#<em>SingleProcessDataLoaderIter.</em>…</td>
<td>0.64%</td>
<td>167.173ms</td>
<td>1.34%</td>
<td>349.861ms</td>
<td>9.207ms</td>
<td>38</td>
</tr>
<tr>
<td>aten::randperm</td>
<td>0.00%</td>
<td>436.807us</td>
<td>0.00%</td>
<td>876.528us</td>
<td>219.132us</td>
<td>4</td>
</tr>
<tr>
<td>aten::scalar_tensor</td>
<td>0.00%</td>
<td>7.450us</td>
<td>0.00%</td>
<td>7.450us</td>
<td>3.725us</td>
<td>2</td>
</tr>
<tr>
<td>aten::resize_</td>
<td>0.01%</td>
<td>1.711ms</td>
<td>0.01%</td>
<td>1.711ms</td>
<td>2.554us</td>
<td>670</td>
</tr>
<tr>
<td>aten::resolve_conj</td>
<td>0.01%</td>
<td>2.982ms</td>
<td>0.01%</td>
<td>2.982ms</td>
<td>0.326us</td>
<td>9141</td>
</tr>
</tbody>
</table>
</div><p><em>Self CPU time total: 26.178s</em></p>
<hr>
<p><strong>GPU/CUDA Runtime</strong></p>
<div class="md-table">
<table>
<thead>
<tr>
<th>Name</th>
<th>Self CPU %</th>
<th>Self CPU</th>
<th>CPU total %</th>
<th>CPU total</th>
<th>CPU time avg</th>
<th>Self CUDA</th>
<th>Self CUDA %</th>
<th>CUDA total</th>
<th>CUDA time avg</th>
<th># of Calls</th>
</tr>
</thead>
<tbody>
<tr>
<td>model_inference</td>
<td>0.00%</td>
<td>0.00us</td>
<td>0.00%</td>
<td>0.00us</td>
<td>0.00us</td>
<td><strong>1439.411s</strong></td>
<td>95.27%</td>
<td><strong>1439.411s</strong></td>
<td><strong>719.705s</strong></td>
<td>2</td>
</tr>
<tr>
<td>GraphLowering.run (dynamo_timed)</td>
<td>0.00%</td>
<td>0.00us</td>
<td>0.00%</td>
<td>0.00us</td>
<td>0.00us</td>
<td><strong>59.987s</strong></td>
<td>3.97%</td>
<td><strong>59.987s</strong></td>
<td>810.641ms</td>
<td>74</td>
</tr>
<tr>
<td>CachingAutotuner.benchmark_all_configs (dynamo_timed…</td>
<td>0.00%</td>
<td>0.00us</td>
<td>0.00%</td>
<td>0.00us</td>
<td>0.00us</td>
<td>5.193s</td>
<td>0.34%</td>
<td>5.193s</td>
<td>144.253ms</td>
<td>36</td>
</tr>
<tr>
<td>aten::fill_</td>
<td>0.13%</td>
<td>950.923ms</td>
<td>0.17%</td>
<td>1.249s</td>
<td>28.885us</td>
<td>3.996s</td>
<td>0.26%</td>
<td>3.996s</td>
<td>92.384us</td>
<td>43249</td>
</tr>
<tr>
<td>aten::zero_</td>
<td>0.13%</td>
<td>947.515ms</td>
<td>0.30%</td>
<td>2.238s</td>
<td>49.680us</td>
<td>0.000us</td>
<td>0.00%</td>
<td>3.994s</td>
<td>88.611us</td>
<td>45049</td>
</tr>
<tr>
<td>void at::native::vectorized_elementwise_kernel&lt;4, at…</td>
<td>0.00%</td>
<td>0.00us</td>
<td>0.00%</td>
<td>0.00us</td>
<td>0.00us</td>
<td>3.956s</td>
<td>0.26%</td>
<td>3.956s</td>
<td>287.800us</td>
<td>13747</td>
</tr>
<tr>
<td>CachingAutotuner.benchmark_all_configs (dynamo_timed…</td>
<td>0.36%</td>
<td>2.618s</td>
<td>0.71%</td>
<td>5.23s</td>
<td>145.287ms</td>
<td>0.000us</td>
<td>0.00%</td>
<td>3.737s</td>
<td>103.808ms</td>
<td>36</td>
</tr>
<tr>
<td>model_inference</td>
<td>0.43%</td>
<td>3.184s</td>
<td>99.58%</td>
<td><strong>731.765s</strong></td>
<td><strong>731.765s</strong></td>
<td>0.000us</td>
<td>0.00%</td>
<td>2.663s</td>
<td>2.663s</td>
<td>1</td>
</tr>
<tr>
<td>Torch-Compiled Region</td>
<td>0.00%</td>
<td>11.368ms</td>
<td>0.52%</td>
<td>103.364ms</td>
<td>3.824s</td>
<td>0.000us</td>
<td>0.00%</td>
<td>2.298s</td>
<td>62.111ms</td>
<td>37</td>
</tr>
<tr>
<td>CompiledFunction</td>
<td>0.10%</td>
<td>768.567ms</td>
<td>0.52%</td>
<td>103.032ms</td>
<td>3.812s</td>
<td>158.932ms</td>
<td>0.01%</td>
<td>2.298s</td>
<td>62.111ms</td>
<td>37</td>
</tr>
</tbody>
</table>
</div><p>Self CPU time total: 734.872s<br>
Self CUDA time total: 1510.825s</p>
<p>Any insight as to what’s causing this would be greatly appreciated.</p>

mloewen: <p><strong>Update:</strong> I believe I might’ve discovered an error in my environment setup.</p>
<p>I installed Pytorch, targeting CUDA <strong>12.4</strong> via the following:</p>
<blockquote>
<p>conda install pytorch torchvision torchaudio pytorch-cuda=12.4 -c pytorch -c nvidia</p>
</blockquote>
<p>When checking the CUDA version, it shows as <strong>10.1</strong>:</p>
<blockquote>
<p>$ nvcc -V</p>
</blockquote>
<blockquote>
<p>nvcc: NVIDIA (R) Cuda compiler driver<br>
Copyright (c) 2005-2019 NVIDIA Corporation<br>
Built on Sun_Jul_28_19:07:16_PDT_2019<br>
Cuda compilation tools, release 10.1, V10.1.243</p>
</blockquote>
<p>Checking CUDA versions installed on the machine shows only <strong>12.2+</strong> versions:</p>
<blockquote>
<p>$ ls -l /usr/local | grep cuda</p>
</blockquote>
<blockquote>
<p>lrwxrwxrwx  1 root root   22 Aug 20 15:39 cuda → /etc/alternatives/cuda<br>
lrwxrwxrwx  1 root root   25 May  1 15:09 cuda-12 → /etc/alternatives/cuda-12<br>
drwxr-xr-x 16 root root 4096 Aug 20 15:25 cuda-12.2<br>
drwxr-xr-x 15 root root 4096 May  3 14:34 cuda-12.3<br>
drwxr-xr-x 15 root root 4096 May  1 15:09 cuda-12.4</p>
</blockquote>
<p>So, I have installed Pytorch targeting CUDA 12.4 and have CUDA 12.2, 12.3, and 12.4 installed on the machine, but the environment is pointing towards CUDA 10.1, which is potentially the problem…</p>
<p>Has anyone experienced this before?</p>

ptrblck: <p>A few issues stand out in your profiling:</p>
<ul>
<li>You are profiling the actual compilation step of <code>torch.compile</code> which is showing a long duration,</li>
<li><code>autocast</code> is not enabled but I assume it’s neither enabled on the CPU (<code>bfloat16</code> is also not supported on your Volta GPU)</li>
<li>Your locally installed CUDA toolkit won’t be used unless you build PyTorch from source or a custom CUDA extension.</li>
</ul>

mloewen: <p>Thank you for the suggestions ptrblck.</p>
<p>I am running the Profiler outside the minibatch_loader as such:</p>
<blockquote>
<pre><code># profile torch run
with profile(activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA], record_shapes=True) as prof:
    with record_function("model_inference"):

        # cycle all minibatches
        for batch, data in enumerate(batch_loader, 0):
            # training data
            x_, y_ = data[0].float(), data[1].float()
            x_, y_ = x_.to(device), y_.to(device)
            with torch.autocast(device_type=device, dtype=torch.float32, enabled=True):
                logits, loss = model(x_, y_)

            loss = loss / batches_per_epoch
            if train:
                loss.backward()  # accumulate the loss

            # accumulate
            loss_accum += loss.detach()
            logits_accum = torch.cat((logits_accum, logits.detach()), dim=0)
            y_accum = torch.cat((y_accum, y_), dim=0)

print(prof.key_averages().table(sort_by="cuda_time_total", row_limit=10))
</code></pre>
</blockquote>
<p>So, it cycles through an entire epoch of minibatches one time before printing those results. I have messed around with doing inside each minibatch run, but the results are pretty much identical, just scaled down by n_batches.</p>
<p>I’ve subsequently changed the autocast dtype and enable params as suggested:</p>
<blockquote>
<p>with torch.autocast(device_type=device, dtype=torch.float32, enabled=True):</p>
</blockquote>
<p>and have matrix multiplication precision set to <em>medium</em>.</p>
<blockquote>
<p>torch.set_float32_matmul_precision(“medium”)</p>
</blockquote>
<p>I’m not very familiar with the last suggestion of building from source and am working to figure that out… If anyone knows of any step-by-step guides for setting up in a conda environment on a Linux server, that would be greatly appreciated.</p>

ptrblck: <aside class="quote no-group" data-username="mloewen" data-post="4" data-topic="209774">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mloewen/48/72832_2.png" class="avatar"> mloewen:</div>
<blockquote>
<p>I’m not very familiar with the last suggestion of building from source and am working to figure that out…</p>
</blockquote>
</aside>
<p>You don’t need to build from source and the PyTorch binaries are fine unless you have a valid use case to use a locally installed CUDA toolkit.<br>
Add warmup iterations to get rid of the compilation step and to profile the actual workload afterwards.<br>
Changing the matmul precision won’t do anything on your device as it’s enabling <code>TF32</code> on Ampere+ GPUs.</p>

mloewen: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> - I did resolve my issue; the solution was co-linear to your recommendations. Thank you for taking the time to reply with suggestions.</p>
<p>After completely wiping my OS and starting with a fresh Ubuntu 22.04 install, Nvidia drivers, and CUDA, etc… I ultimately had two problems. Firstly, I wasn’t careful with the dtypes of my dataset flowing into the model and, secondly, torch.compile() is not working, which is unfortunate. I’ll break this down for others to follow:</p>
<p><strong>Issue <span class="hashtag-raw">#1</span> - dtypes:</strong> I am using an Nvidia V100 on a MSFT Azure VM. The V100 is older hardware that supports only FP16 precision on its tensor cores. I found this out by searching for many days through GitHub forums to find the following [source: <a href="https://github.com/artidoro/qlora/issues/71" class="inline-onebox" rel="noopener nofollow ugc">V100 can not supprt load_in_4bit and fp16? · Issue #71 · artidoro/qlora · GitHub</a>] (hopefully Nvidia documentation improves). Anyways, I originally had FP64 and Int dtypes flowing through my dataloader. Once I fixed the dtypes to FP16, the V100 GPU was now able to match the speed of a CPU.</p>
<p>Note: AMP (Automatic Mixed Precision) was not able to fix this problem for some reason…</p>
<p><strong>Issue <span class="hashtag-raw">#2</span> - torch.compile:</strong> After solving the above issue, a new runtime error popped up.</p>
<blockquote>
<p>torch/fx/experimental/symbolic_shapes.py:4449] [0/0_1] xindex is not in var_ranges, defaulting to unknown range.</p>
</blockquote>
<p>I had no idea how to interpret this, but “experimental/symbolic_shapes” suggested that this might have something to due with pre-compiling the model. There are a handful of GitHub discussions/issues reporting this (e.g. <a href="https://github.com/MIC-DKFZ/nnUNet/issues/2418" class="inline-onebox" rel="noopener nofollow ugc">q0 is not in var_ranges,z0 is not in var_ranges,x0 is not in var_ranges, · Issue #2418 · MIC-DKFZ/nnUNet · GitHub</a>) so I simply commented-out the following before runtime:</p>
<blockquote>
<pre><code>model = torch.compile(model, mode="reduce-overhead", fullgraph=True)
</code></pre>
</blockquote>
<p>Success! It now works as intended.</p>
<p>I don’t know why torch.compiler doesn’t like my model. It’s pretty similar to GPT3, although I do have some custom layers and configurations, all built within the Pytorch ecosystem.</p>
<p>My model is now able to run substantially faster than a CPU. What used to take 3 min per epoch, now takes less than a dozen seconds.</p>
<p>Thank you again for your dedication to Pytorch and helping others.</p>