Enable torch.compile only for backward

Chengji_Yao: <p>Hi,</p>
<p>In our use cases, I’d like to optimize the backward graph only and leave forward graph running in eager mode. Is it possible to add an option in torch.compile like this?</p>
<pre><code class="lang-auto">torch._dynamo.config.compiled_autograd = True
@torch.compile(bwd_only=True, ...)
class MyModule(nn.Module):
    ....
</code></pre>

bdhirsh: <p>We actually have an option for compiling only the backward graph - compiled autograd. It’s still experimental, but you can try it out with something like:</p>
<pre><code class="lang-auto"># run the fw in eager (or with compile)
out = model(inp)
loss = loss_fn(out, ...)

# take the entire backward graph, and compile it
with torch._dynamo.utils.maybe_enable_compiled_autograd(True, fullgraph=True, dynamic=False):
    loss.backward()
</code></pre>
<p>Ed also has a podcast on it if you want to learn more about it: <a href="https://pytorch-dev-podcast.simplecast.com/episodes/compiled-autograd-TCcEyBRZ" rel="noopener nofollow ugc">https://pytorch-dev-podcast.simplecast.com/episodes/compiled-autograd-TCcEyBRZ</a></p>

Chengji_Yao: <p><a class="mention" href="/u/bdhirsh">@bdhirsh</a> , thanks for your reply!</p>
<p>A follow up question, do you know how to compile the backward of a single nn.Module, not the entire backward graph?</p>

bdhirsh: <p><code>torch.compil</code>ing that individual module will be easiest, although it will result in your forward also getting compiled.</p>
<p>Out of curiosity - what use case do you have that requires wanting to keep everything in eager mode, except a specific region of the backward?</p>

Chengji_Yao: <aside class="quote no-group" data-username="bdhirsh" data-post="4" data-topic="211402">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bdhirsh/48/74038_2.png" class="avatar"> bdhirsh:</div>
<blockquote>
<p>what use case do you have that requires wanting to keep everything in eager mode, except a specific region of the backward?</p>
</blockquote>
</aside>
<p>It’s for LLM training. In our code base, <code>torch.compile</code> will fail for an individual Transformer module since our forward logic is too complex. While what we really want to optimize is the backward of Transformer block since there are plenty of opportunities to overlap communication and computation in the backward. We are considering to implement it automatically. And I believe capturing the backward graph only will be easier than capturing the forward graph.</p>

bdhirsh: <p>Got it. I can’t give much more specific advice without more detail, but a few things you might find useful are:</p>
<p>Custom ops: if there is a region of your forward that is well optimized (too complicated to naively capture with torch.compile, and compile won’t be able to speed it up much anyway, you can wrap it in a custom operator. Torch.compile will compile everything around it, but treat it as a black box that gets invoked at runtime. <a href="https://pytorch.org/tutorials/advanced/python_custom_ops.html" class="inline-onebox" rel="noopener nofollow ugc">Python Custom Operators — PyTorch Tutorials 2.4.0+cu121 documentation</a></p>
<p>Functional collectives. Compile has some support for compiling comms operator’s, if you want to let the compiler try optimizing them. <a href="https://github.com/pytorch/pytorch/issues/93173" class="inline-onebox" rel="noopener nofollow ugc">[RFC] PT2-Friendly Traceable, Functional Collective Communication APIs · Issue #93173 · pytorch/pytorch · GitHub</a></p>

Chengji_Yao: <p>Sometimes even python custom operators can not solve the problem. A good example is <a href="https://github.com/NVIDIA/Megatron-LM/blob/main/megatron/legacy/model/transformer.py#L407" class="inline-onebox" rel="noopener nofollow ugc">Megatron-LM/megatron/legacy/model/transformer.py at main · NVIDIA/Megatron-LM · GitHub</a>, where I tried to wrap the code into a python custom op.</p>
<pre><code class="lang-auto">with tensor_parallel.get_cuda_rng_tracker().fork():
    attention_probs = self.attention_dropout(attention_probs)
</code></pre>
<p>In this way, it can be successfully captured by dynamo, but got errors when running the model. Seems that dynamo can not handle random state well.</p>
<p>While in backward, the rng code will not appear in autograd backward graph, only the backward of dropout will. That’s why I believe only capturing backward is easier than capturing both forward and backward graph.</p>