Compiling the model results in 20X slow down

username: mer
cooked: <p>Hi,<br>
I’m using the default settings for model compilation.</p>
<p>In my case, compiling the model results in a 20X slow down. I left two models running (one compiled and one not), and the results are:</p>
<p>compiled: 873 steps in 8 hours<br>
not-compiled: 16 256 steps in 8 hours</p>
<p>Each time during a forward, I’m passing tensor of the same dimensions exactly (BS x padded-len).</p>
<p>For the compiled model, the first step of the first batch takes A LOT of time [5 mins for the compilation time?].</p>
<p>Compiled model:<br>
2023/08/03 23:55:33	Epoch 0: 0%| | 1/6103 [04:53&lt;497:03:28, 293.25s/it]<br>
2023/08/03 23:56:01	Epoch 0: 0%| | 2/6103 [05:20&lt;271:54:04, 160.44s/it, v_num=-332]<br>
2023/08/03 23:56:28	Epoch 0: 0%| | 3/6103 [05:47&lt;196:25:00, 115.92s/it, v_num=-332]<br>
2023/08/03 23:57:00	Epoch 0: 0%| | 4/6103 [06:19&lt;160:49:52, 94.93s/it, v_num=-332]</p>
<p>Not-compiled model:<br>
2023/08/03 23:57:41	Epoch 0: 0%| | 1/6103 [00:07&lt;12:09:16, 7.17s/it, v_num=-334]<br>
2023/08/03 23:57:42	Epoch 0: 0%| | 2/6103 [00:08&lt;7:23:44, 4.36s/it, v_num=-334]<br>
2023/08/03 23:57:44	Epoch 0: 0%| | 3/6103 [00:10&lt;5:48:41, 3.43s/it, v_num=-334]<br>
2023/08/03 23:57:45	Epoch 0: 0%| | 4/6103 [00:11&lt;5:00:26, 2.96s/it, v_num=-334]<br>
2023/08/03 23:57:47	Epoch 0: 0%| | 5/6103 [00:13&lt;4:32:01, 2.68s/it, v_num=-334]<br>
2023/08/03 23:57:48	Epoch 0: 0%| | 6/6103 [00:14&lt;4:12:42, 2.49s/it, v_num=-334]<br>
2023/08/03 23:57:50	Epoch 0: 0%| | 7/6103 [00:16&lt;3:58:51, 2.35s/it, v_num=-334]</p>
<p>Hardware for the runs:<br>
V100 32GB, 1 GPU</p>

username: thorstenwagner
cooked: <p>Hi <a class="mention" href="/u/mer">@mer</a>,</p>
<p>I also had some problems with model compilation. Most of them got solved by using the latest nightly. Could you try it?</p>
<p>Best,<br>
Thorsten</p>

username: mer
cooked: <p>Sure, I’ll try it and post an update in about an hour.</p>
<p>Edit: it might take a little longer <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=12" title=":sweat_smile:" class="emoji" alt=":sweat_smile:" loading="lazy" width="20" height="20"></p>
<pre><code class="lang-auto">Installing collected packages: pytorch-triton, torch, torchvision, torchaudio
ERROR: Could not install packages due to an OSError: [Errno 38] Function not implemented: '/usr/local/lib/python3.10/site-packages/triton/__init__.py'
</code></pre>

username: mer
cooked: <p><a class="mention" href="/u/thorstenwagner">@thorstenwagner</a></p>
<p>My torch packages:</p>
<pre><code class="lang-auto">Singularity&gt; pip freeze | grep torch
pytorch-lightning==2.0.6
pytorch-metric-learning==2.3.0
pytorch-ranger==0.1.1
pytorch-triton==2.1.0+440fd1bf20
torch==2.1.0.dev20230620+cu121
torch-optimizer==0.3.0
torchaudio==2.1.0.dev20230620+cu121
torchmetrics==0.11.4
torchvision==0.16.0.dev20230620+cu121
</code></pre>
<p>Results - all run for more than 1 hour on an A100 40GB:<br>
compiled -</p>
<pre><code class="lang-auto">Epoch 0: 2%|▏ | 135/6103 [1:14:59&lt;55:15:02, 33.33s/it, v_num=-348]
</code></pre>
<p>compiled, dynamic=True -</p>
<pre><code class="lang-auto">Epoch 0: 2%|▏ | 107/6103 [1:14:52&lt;69:55:47, 41.99s/it, v_num=-350]
</code></pre>
<p>not-compiled -</p>
<pre><code class="lang-auto">Epoch 1: 87%|████████▋ | 5337/6103 [34:55&lt;05:00, 2.55it/s, v_num=-349]
</code></pre>

username: mer
cooked: <p>Here is my env:</p>
<pre><code class="lang-auto">Singularity&gt; python -m torch.utils.collect_env
Collecting environment information...
PyTorch version: 2.1.0.dev20230620+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Debian GNU/Linux 11 (bullseye) (x86_64)
GCC version: (Debian 10.2.1-6) 10.2.1 20210110
Clang version: Could not collect
CMake version: version 3.27.0
Libc version: glibc-2.31

Python version: 3.10.10 (main, Mar 23 2023, 03:59:34) [GCC 10.2.1 20210110] (64-bit runtime)
Python platform: Linux-5.14.0-162.18.1.el9_1.x86_64-x86_64-with-glibc2.31
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA A100-SXM4-40GB
Nvidia driver version: 525.85.12
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True
</code></pre>

username: thorstenwagner
cooked: <p>Do you compile it ‘reduce-overhead’ mode?</p>
<p>Like</p>
<pre><code class="lang-auto">model = torch.compile(model, mode="reduce-overhead")
</code></pre>

username: mer
cooked: <p>After about 2 hours:</p>
<pre><code class="lang-auto">Epoch 0: 6%|▌ | 359/6103 [2:10:40&lt;34:50:47, 21.84s/it, v_num=-544] - reduce-overhead
Epoch 0: 4%|▍ | 255/6103 [2:10:39&lt;49:56:24, 30.74s/it, v_num=-543] - default
Epoch 0: 6%|▌ | 378/6103 [2:10:39&lt;32:58:47, 20.74s/it, v_num=-542] - max-autotune
</code></pre>
<p>It’s nowhere near not compiling : /</p>

username: ptrblck
cooked: <aside class="quote no-group quote-modified" data-username="mer" data-post="4" data-topic="185693">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/65b543/48.png" class="avatar"> mer:</div>
<blockquote>
<p>My torch packages:</p>
<pre><code class="lang-auto">pytorch-triton==2.1.0+440fd1bf20
torch==2.1.0.dev20230620+cu121
</code></pre>
</blockquote>
</aside>
<p>You are still on an older nightly release from June, so could you try to update to the latest nightly release or are you still running into the installation issues?</p>

username: mer
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>,<br>
First of all, thanks for all the responses to all of the threads, not only this one. You have helped me and others using Pytorch soooo much.</p>
<p>I have updated the packages. Previously, I just used</p>
<pre><code class="lang-auto">pip3 install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu121
</code></pre>
<p>And it downloaded those versions. I somehow have done it again, to the latest versions, and the issue seems to be resolved.</p>
<p>After 5 mins, the “not-compiled” was the fastest, but after 35 mins the “max-autotune” was actually about 15% faster. I’ll try to post an update later.</p>
<p>Thanks so much.</p>

username: marksaroufim
cooked: <aside class="quote no-group" data-username="mer" data-post="1" data-topic="185693">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/65b543/48.png" class="avatar"> mer:</div>
<blockquote>
<p>For the compiled model, the first step of the first batch takes A LOT of time [5 mins for the compilation time?].</p>
</blockquote>
</aside>
<p>compilation happens when after you run <code>opt_m = torch.compile()</code>, when you run <code>opt_m(batch)</code> - the naming is confusing, compile should be called a JIT but that was already taken XD</p>