Tried basic torch.compile but resul is counter-intuitive

jury_duty: <p>I ran most basic example here (<a href="https://pytorch.org/tutorials/intermediate/torch_compile_tutorial.html#basic-usage" class="inline-onebox" rel="noopener nofollow ugc">torch.compile Tutorial — PyTorch Tutorials 2.1.1+cu121 documentation</a>) but modified to measure execution time of both unoptimized and optimized one but optimized version seems to run much slower. I ran 5 times just to account for warmup time still worse. weird, confused. Can anyone explain?<br>
Does optimization by torch.compile means speed or somehing else?</p>
<blockquote>
<p>import torch<br>
import time</p>
<p>def foo(x, y):<br>
a = torch.sin(x)<br>
b = torch.cos(y)<br>
return a + b<br>
opt_foo1 = torch.compile(foo)</p>
<p>for i in range(0,5):<br>
t1_foo = time.time_ns()<br>
foo(torch.randn(10, 10), torch.randn(10, 10))<br>
t2_foo = time.time_ns()<br>
print("time: unoptimized:   ", t2_foo-t1_foo)</p>
<p>for i in range(0,5):<br>
t1_foo1 = time.time_ns()<br>
opt_foo1(torch.randn(10, 10), torch.randn(10, 10))<br>
t2_foo1 = time.time_ns()<br>
print("time: optimized:     ", t2_foo1-t1_foo1)</p>
</blockquote>
<p>output:</p>
<blockquote>
<p>python3 ex-1-modded.py<br>
time: unoptimized:    24900263<br>
time: unoptimized:    34249<br>
time: unoptimized:    30179<br>
time: unoptimized:    30480<br>
time: unoptimized:    26619<br>
time: optimized:      12733648173<br>
time: optimized:      129398<br>
time: optimized:      61689<br>
time: optimized:      60659<br>
time: optimized:      68889</p>
</blockquote>

jury_duty: <p>Apparently later examples work and optimized well, not sure what i did wrong in my example:</p>
<p>import torch</p>
<h1><a name="returns-the-result-of-running-fn-and-the-time-it-took-for-fn-to-run-1" class="anchor" href="#returns-the-result-of-running-fn-and-the-time-it-took-for-fn-to-run-1"></a>Returns the result of running <code>fn()</code> and the time it took for <code>fn()</code> to run,</h1>
<h1><a name="in-seconds-we-use-cuda-events-and-synchronization-for-the-most-accurate-2" class="anchor" href="#in-seconds-we-use-cuda-events-and-synchronization-for-the-most-accurate-2"></a>in seconds. We use CUDA events and synchronization for the most accurate</h1>
<h1><a name="measurements-3" class="anchor" href="#measurements-3"></a>measurements.</h1>
<p>def timed(fn):<br>
start = torch.cuda.Event(enable_timing=True)<br>
end = torch.cuda.Event(enable_timing=True)<br>
start.record()<br>
result = fn()<br>
end.record()<br>
torch.cuda.synchronize()<br>
return result, start.elapsed_time(end) / 1000</p>
<h1><a name="generates-random-input-and-targets-data-for-the-model-where-b-is-4" class="anchor" href="#generates-random-input-and-targets-data-for-the-model-where-b-is-4"></a>Generates random input and targets data for the model, where <code>b</code> is</h1>
<h1><a name="batch-size-5" class="anchor" href="#batch-size-5"></a>batch size.</h1>
<p>def generate_data(b):<br>
return (<br>
torch.randn(b, 3, 128, 128).to(torch.float32).cuda(),<br>
torch.randint(1000, (b,)).cuda(),<br>
)</p>
<p>N_ITERS = 10</p>
<p>from torchvision.models import densenet121<br>
def init_model():<br>
return densenet121().to(torch.float32).cuda()</p>
<p>model = init_model()</p>
<h1><a name="reset-since-we-are-using-a-different-mode-6" class="anchor" href="#reset-since-we-are-using-a-different-mode-6"></a>Reset since we are using a different mode.</h1>
<p>import torch._dynamo<br>
torch._dynamo.reset()</p>
<p>model_opt = torch.compile(model, mode=“reduce-overhead”)</p>
<p>inp = generate_data(16)[0]</p>
<p>eager_times = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
for i in range(N_ITERS):<br>
inp = generate_data(16)[0]<br>
with torch.no_grad():<br>
_, eager_time = timed(lambda: model(inp))<br>
eager_times.append(eager_time)<br>
print(f"eager eval time {i}: {eager_time}")</p>
<p>print(“~” * 10)</p>
<p>compile_times = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
for i in range(N_ITERS):<br>
inp = generate_data(16)[0]<br>
with torch.no_grad():<br>
_, compile_time = timed(lambda: model_opt(inp))<br>
compile_times.append(compile_time)<br>
print(f"compile eval time {i}: {compile_time}“)<br>
print(”~" * 10)</p>
<p>import numpy as np<br>
eager_med = np.median(eager_times)<br>
compile_med = np.median(compile_times)<br>
speedup = eager_med / compile_med<br>
assert(speedup &gt; 1)<br>
print(f"(eval) eager median: {eager_med}, compile median: {compile_med}, speedup: {speedup}x")<br>
print(“~” * 10)</p>
<hr>
<p>python3 ex-3-original-2.py<br>
eager eval time 0: 1.8496431884765625<br>
eager eval time 1: 0.02884818458557129<br>
eager eval time 2: 0.026322099685668947<br>
eager eval time 3: 0.026348180770874023<br>
eager eval time 4: 0.02663505935668945<br>
eager eval time 5: 0.02669969940185547<br>
eager eval time 6: 0.026537784576416014<br>
eager eval time 7: 0.033738910675048826<br>
eager eval time 8: 0.027683704376220704<br>
eager eval time 9: 0.026418901443481444</p>
<pre><code class="lang-auto">/usr/local/lib/python3.10/dist-packages/torch/_inductor/compile_fx.py:135: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
  warnings.warn(
compile eval time 0: 78.0921796875
compile eval time 1: 0.785571533203125
compile eval time 2: 0.0111019287109375
compile eval time 3: 0.010863529205322266
compile eval time 4: 0.010895689010620118
compile eval time 5: 0.01093185043334961a
compile eval time 6: 0.011064648628234863
compile eval time 7: 0.01088080883026123
compile eval time 8: 0.011026410102844239
compile eval time 9: 0.010890089035034179
</code></pre>
<p>(eval) eager median: 0.02666737937927246, compile median: 0.010979130268096923, speedup: 2.4289154721811013x</p>

soulitzer: <p>Your function in the first function is quite small. When using compile there is always some constant overhead during runtime of guard evaluation etc, so compile tends to do better when you have larger programs as that would amortize away some of that overhead.</p>