[Distributed w/ TorchTitan] Introducing Async Tensor Parallelism in PyTorch

username: yifuwang
cooked: <p>with <a href="https://dev-discuss.pytorch.org/u/Chillee/summary" rel="noopener nofollow ugc">Horace He</a>, <a href="https://discuss.pytorch.org/u/lessw2020/summary">Less Wright</a>, <a href="https://discuss.pytorch.org/u/lcw/summary">Luca Wehrstedt</a>, <a href="https://discuss.pytorch.org/u/wanchaol/summary">Tianyu Liu</a>, <a href="https://dev-discuss.pytorch.org/u/wanchaol/summary" rel="noopener nofollow ugc">Wanchao Liang</a></p>
<h2><a name="p-454286-tldr-1" class="anchor" href="#p-454286-tldr-1"></a>TL;DR</h2>
<ul>
<li>We implemented experimental async tensor parallelism support in PyTorch.</li>
<li>We integrated it in TorchTitan and observed:
<ul>
<li>Up to ~29% forward pass speedup and ~8% E2E speedup in Llama3 7B.</li>
<li>Up to ~20% forward pass speedup and ~8% E2E speedup in Llama3 70B.</li>
</ul>
</li>
<li>We briefly discuss the performance challenges and the solutions we devised.</li>
</ul>
<h2><a name="p-454286-distributed-with-torchtitan-2" class="anchor" href="#p-454286-distributed-with-torchtitan-2"></a>Distributed with TorchTitan</h2>
<p>The GitHub repository <a href="https://github.com/pytorch/torchtitan" rel="noopener nofollow ugc">torchtitan</a> is a proof of concept for large-scale LLM training using native PyTorch, designed to be easy to understand, use, and extend for different training purposes, supporting multi-dimensional parallelisms with modular components. In this series of topics, we introduce the latest PyTorch features for distributed training enabled in TorchTitan.</p>
<ul>
<li>Topic 1: <a href="https://discuss.pytorch.org/t/distributed-w-torchtitan-enabling-float8-all-gather-in-fsdp2/209323">Enabling Float8 All-Gather in FSDP2</a></li>
<li> → Topic 2: Introducing Async Tensor Parallelism in PyTorch</li>
</ul>
<h2><a name="p-454286-tensor-parallelism-3" class="anchor" href="#p-454286-tensor-parallelism-3"></a>Tensor Parallelism</h2>
<p>Tensor parallelism (TP) is a widely used model parallelism technique. Unlike data parallelism, which is limited to sharding computations across the batch dimension, TP further distributes the computation along the feature dimension, allowing multiple GPUs to process the same sample simultaneously. This characteristic makes TP crucial for large-scale LLM training, allowing the number of devices to scale beyond the global batch size.</p>
<p>                    <a href="https://raw.githubusercontent.com/pytorch-labs/applied-ai/refs/heads/main/assets/images/dev-discuss-asynctp/Figure_1.png" target="_blank" rel="noopener nofollow ugc" class="onebox">
            <img src="https://raw.githubusercontent.com/pytorch-labs/applied-ai/refs/heads/main/assets/images/dev-discuss-asynctp/Figure_1.png" width="690" height="156">
          </a>

<br>
<em>Figure 1: TP applied to a two-layer FFN sharded across 2 devices</em></p>
<p>As a brief recap, the diagram illustrates TP applied to a two-layer FFN sharded across 2 devices. We begin with row-wise sharded inputs <code>[X0, X1]</code>, column-wise sharded linear weights <code>[A0, A1]</code>, and row-wise sharded linear weights [B0, B1]. First, an all-gather is performed on <code>[X0, X1]</code> to produce the unsharded input <code>X</code>. Then, <code>X @ A0 @ B0</code> and <code>X @ A1 @ B1</code> are computed independently on each device, with activations remaining sharded. The resulting partial sums of the unsharded output are then combined using a reduce-scatter to form the final sharded output.</p>
<p>This approach efficiently minimizes communication volume by keeping the activations sharded as long as possible. However, the communication still poses efficiency challenges because it is exposed. Async tensor parallelism is an optimization designed to address this issue.</p>
<h2><a name="p-454286-async-tensor-parallelism-4" class="anchor" href="#p-454286-async-tensor-parallelism-4"></a>Async Tensor Parallelism</h2>
<p>The concept of async tensor parallelism (async-TP [<a href="#note1">1</a>] was, to our knowledge, first introduced in the paper <a href="https://arxiv.org/abs/2105.05720" rel="noopener nofollow ugc">Breaking the Computation and Communication Abstraction Barrier in Distributed Machine Learning Workloads</a>, although there have been several parallel efforts, including <a href="https://dl.acm.org/doi/abs/10.1145/3567955.3567959" rel="noopener nofollow ugc">Wang et al. 2022.</a> and <a href="https://arxiv.org/abs/2406.06858" rel="noopener nofollow ugc">Chang et al. 2024</a>. The key insight is that by decomposing dependent communication and computation operators, we can create overlapping opportunities that would otherwise be unachievable.</p>
<p>                    <a href="https://raw.githubusercontent.com/pytorch-labs/applied-ai/refs/heads/main/assets/images/dev-discuss-asynctp/Figure_2.png" target="_blank" rel="noopener nofollow ugc" class="onebox">
            <img src="https://raw.githubusercontent.com/pytorch-labs/applied-ai/refs/heads/main/assets/images/dev-discuss-asynctp/Figure_2.png" width="690" height="436">
          </a>

<br>
<em>Figure 2: Async-TP applied to an all-gather followed by a matmul</em></p>
<p>The diagram from Wang et al. illustrates how this technique can be applied to an all-gather followed by a matmul. The all-gather is decomposed into send and recv operations, while the matmul is divided into sub-matmuls. With this decomposition, it becomes possible to compute one sub-matmul while simultaneously transfering the data required for the next sub-matmul, effectively hiding communication latency.</p>
<h2><a name="p-454286-performance-challenges-5" class="anchor" href="#p-454286-performance-challenges-5"></a>Performance Challenges</h2>
<p>Although the concept of async-TP is straightforward in theory, achieving a performant CUDA implementation presented several challenges. In this section, we’ll explore these challenges and discuss the approaches we employed to address them.</p>
<p><strong>Acknowledgment:</strong> Many of these challenges were initially explored by <a href="https://discuss.pytorch.org/u/lcw/summary">Luca Wehrstedt</a>. The async-TP implementation in PyTorch drew significant inspiration from his async-TP work in <a href="https://github.com/facebookresearch/xformers" rel="noopener nofollow ugc">xformers</a>.</p>
<h4><a name="p-454286-communication-overhead-6" class="anchor" href="#p-454286-communication-overhead-6"></a>Communication Overhead</h4>
<p>When decomposing communication, it might be tempting to use NCCL send/recv due to their accessibility. However, NCCL send/recv has certain characteristics that make them less ideal for async-TP:</p>
<ul>
<li>
<p><strong>Contention between overlapped computation and communication</strong> - while it’s commonly believed that computation and communication are two resources that can be utilized independently, in reality, their independence is nuanced, and contention does occur. In intra-node settings (most common for TP), NCCL send/recv kernels utilize SMs to move data across NVLink, reducing the number of SMs available for the overlapping matmul kernels and slows them down. Interestingly, the observed slowdown can exceed the percentage of resources consumed by the communication kernels. Since cuBLAS attempts to select kernels that execute in full waves, the SMs taken by the communication kernel can tip the balance, causing the matmul kernels to execute an extra wave.</p>
</li>
<li>
<p><strong>Two-way synchronization</strong> - NCCL send/recv kernels perform two-sided synchronization, meaning that both the sender and receiver are blocked until the operation is complete. This approach is not always optimal for data transfers within intra-op parallelism. Depending on the scenario, it may be preferable to perform a single synchronization for multiple data transfers or to choose between pushing data to or pulling data from remote peers.</p>
</li>
</ul>
<p>Fortunately, we can avoid the previously mentioned downsides by leveraging CUDA’s P2P mechanism. The mechanism allows a device to access memory allocated on a peer device by mapping it into its virtual memory address space. This mechanism enables memory operations (load/store/atomic, etc.) to be executed over NVLink [<a href="#note2">2</a>]. Moreover, when transferring contiguous data to or from peer devices via cudaMemcpyAsync, the operation is handled by copy engines [<a href="#note3">3</a>] and does not require any SMs, thereby avoiding the contention issues discussed earlier [<a href="#note4">4</a>].</p>
<p>To leverage this mechanism for async-TP and similar use cases in the future, we developed an experimental abstraction called <a href="https://github.com/pytorch/pytorch/blob/main/torch/csrc/distributed/c10d/SymmetricMemory.hpp" rel="noopener nofollow ugc">SymmetricMemory</a>. It conceptually represents a buffer symmetrically allocated across a group of devices, providing each GPU with access to all corresponding buffers on its peers via virtual memory/multicast addresses. Direct interaction with SymmetricMemory is not necessary for using async-TP, but users can leverage it to create custom fine-grained, intra-node/intra-op optimizations similar to async-TP.</p>
<h4><a name="p-454286-magnified-wave-quantization-inefficiency-due-to-work-decomposition-7" class="anchor" href="#p-454286-magnified-wave-quantization-inefficiency-due-to-work-decomposition-7"></a>Magnified Wave Quantization Inefficiency Due to Work Decomposition</h4>
<p>Tiled matmul kernels execute in waves of SM count. If the final wave consists of only a few tiles, it will take nearly as long to complete as a full wave, leading to what is known as quantization inefficiency. Decomposing a matmul leads to a smaller number of tiles per kernel [<a href="#note5">5</a>] and the combined quantization inefficiency from the decomposed matmuls is likely to exceed that of the original matmul.</p>
<p>                    <a href="https://raw.githubusercontent.com/pytorch-labs/applied-ai/refs/heads/main/assets/images/dev-discuss-asynctp/Figure_3.png" target="_blank" rel="noopener nofollow ugc" class="onebox">
            <img src="https://raw.githubusercontent.com/pytorch-labs/applied-ai/refs/heads/main/assets/images/dev-discuss-asynctp/Figure_3.png" width="690" height="254">
          </a>

<br>
<em>Figure 3: Decomposing a matmul leads to magnified quantization inefficiency</em></p>
<p>To illustrate the issue, let’s examine an all-gather → matmul example. In this example, <code>A</code> is sharded across 4 devices. Without async-TP, A is first all-gathered from the 4 devices, and then <code>A @ B</code> is computed on all devices. With async-TP, <code>A @ B</code> is decomposed into <code>A0 @ B</code>, <code>A1 @ B</code>, <code>A2 @ B</code>, and <code>A3 @ B</code>. An native async-TP implementation would execute the sub-matmuls sequentially in one stream while prefetching the data required for the next sub-matmul on another stream. This approach effectively hides communication latency. However, because the matmul is decomposed into smaller parts, the number of partial waves increases, leading to a longer overall matmul execution time.</p>
<p>                    <a href="https://raw.githubusercontent.com/pytorch-labs/applied-ai/refs/heads/main/assets/images/dev-discuss-asynctp/Figure_4.png" target="_blank" rel="noopener nofollow ugc" class="onebox">
            <img src="https://raw.githubusercontent.com/pytorch-labs/applied-ai/refs/heads/main/assets/images/dev-discuss-asynctp/Figure_4.png" width="689" height="173">
          </a>

<br>
<em>Figure 4: The alternating-stream implementation allows partial waves to overlap with the next sub-matmul</em></p>
<p>To combat the magnified quantization inefficiency, we adopted an alternating-stream approach. Instead of using dedicated streams for computation and communication, we employ two symmetric streams that alternate roles. This method not only allows us to overlap computation with communication but also enables overlapping the partial wave of the current sub-matmul with the next sub-matmul, thereby mitigating the additional quantization inefficiency caused by decomposition.</p>
<p>                    <a href="https://raw.githubusercontent.com/pytorch-labs/applied-ai/refs/heads/main/assets/images/dev-discuss-asynctp/Figure_5.png" target="_blank" rel="noopener nofollow ugc" class="onebox">
            <img src="https://raw.githubusercontent.com/pytorch-labs/applied-ai/refs/heads/main/assets/images/dev-discuss-asynctp/Figure_5.png" width="690" height="181">
          </a>

<br>
<em>Figure 5: Profiling trace of partial waves overlapping with the next sub-matmul</em></p>
<p>          <a href="https://raw.githubusercontent.com/pytorch-labs/applied-ai/refs/heads/main/assets/images/dev-discuss-asynctp/Figure_6.png" target="_blank" rel="noopener nofollow ugc" class="onebox">
            <img src="https://raw.githubusercontent.com/pytorch-labs/applied-ai/refs/heads/main/assets/images/dev-discuss-asynctp/Figure_6.png" width="690" height="224">
          </a>
<br>
<em>Figure 6: Profiling trace comparison between baseline and async-TP</em></p>
<h2><a name="p-454286-e2e-performance-evaluation-8" class="anchor" href="#p-454286-e2e-performance-evaluation-8"></a>E2E Performance Evaluation</h2>
<p>We conducted an E2E performance evaluation on Llama3 8B and 70B with TorchTitan. We observed up to ~29% forward pass speedup and ~8% E2E speedup in Llama3 8B, and up to ~20% forward pass speedup and ~8% E2E speedup in Llama3 70B.</p>
<p>Benchmark configuration:</p>
<ul>
<li>The benchmarks were conducted with 64 H100 GPUs [<a href="#note6">6</a>], each host equipped with 8 GPUs and NVSwitch.</li>
<li>torch.compile was enabled for both the baseline and async-TP configurations.</li>
<li>The models were trained using bf16 precision.</li>
<li>We applied selective activation checkpointing for Llama3 8B and full activation checkpointing for Llama3 70B.</li>
</ul>
<p>                    <a href="https://raw.githubusercontent.com/pytorch-labs/applied-ai/refs/heads/main/assets/images/dev-discuss-asynctp/Figure_7.png" target="_blank" rel="noopener nofollow ugc" class="onebox">
            <img src="https://raw.githubusercontent.com/pytorch-labs/applied-ai/refs/heads/main/assets/images/dev-discuss-asynctp/Figure_7.png" width="690" height="460">
          </a>

<br>
<em>Figure 7: E2E speedup with async-TP</em></p>
<p>                    <a href="https://raw.githubusercontent.com/pytorch-labs/applied-ai/refs/heads/main/assets/images/dev-discuss-asynctp/Figure_8.png" target="_blank" rel="noopener nofollow ugc" class="onebox">
            <img src="https://raw.githubusercontent.com/pytorch-labs/applied-ai/refs/heads/main/assets/images/dev-discuss-asynctp/Figure_8.png" width="690" height="460">
          </a>

<br>
<em>Figure 8: Forward speedup with async-TP</em></p>
<p>                    <a href="https://raw.githubusercontent.com/pytorch-labs/applied-ai/refs/heads/main/assets/images/dev-discuss-asynctp/Figure_9.png" target="_blank" rel="noopener nofollow ugc" class="onebox">
            <img src="https://raw.githubusercontent.com/pytorch-labs/applied-ai/refs/heads/main/assets/images/dev-discuss-asynctp/Figure_9.png" width="690" height="300">
          </a>

<br>
<em>Figure 9: E2E benchmark data</em></p>
<p>We also conducted benchmarks with async-TP on Llama 3.1 405B. You can find the details <a href="https://github.com/pytorch/torchtitan/blob/main/docs/performance.md" rel="noopener nofollow ugc">here</a>.</p>
<h2><a name="p-454286-using-async-tp-in-torchtitan-9" class="anchor" href="#p-454286-using-async-tp-in-torchtitan-9"></a>Using Async-TP in TorchTitan</h2>
<p>The async-TP support is readily integrated into TorchTitan. To enable it, simply supply the <code>--experimental.enable_async_tensor_parallel</code> option when training with tensor parallelism.</p>
<h2><a name="p-454286-using-async-tp-in-pytorch-10" class="anchor" href="#p-454286-using-async-tp-in-pytorch-10"></a>Using Async-TP in PyTorch</h2>
<p>The async-TP support is available in the latest PyTorch nightly builds. You can use it either with torch.compile or directly in eager mode.</p>
<h4><a name="p-454286-using-async-tp-with-torchcompile-11" class="anchor" href="#p-454286-using-async-tp-with-torchcompile-11"></a>Using Async-TP with torch.compile:</h4>
<p>                    <a href="https://raw.githubusercontent.com/pytorch-labs/applied-ai/refs/heads/main/assets/images/dev-discuss-asynctp/Figure_10.png" target="_blank" rel="noopener nofollow ugc" class="onebox">
            <img src="https://raw.githubusercontent.com/pytorch-labs/applied-ai/refs/heads/main/assets/images/dev-discuss-asynctp/Figure_10.png" width="689" height="235">
          </a>

<br>
<em>Figure 10: torch.compile automatically detects TP patterns and rewrites them into async-TP ops</em></p>
<p>torch.compile is currently our recommended method for applying async-TP:</p>
<ul>
<li>It automatically detects TP patterns in your model and rewrites them into async-TP ops, allowing your model to maintain its original structure.</li>
<li>The optimized async-TP implementation requires the inputs to be in a specific layout; otherwise extra copies will occur. torch.compile automatically ensures that upstream ops produce outputs in the desired layout whenever possible.</li>
<li>torch.compile is also capable of detecting situations where an all-gather can be overlapped with multiple matmuls, potentially better hiding communication latency.</li>
</ul>
<p>While these can be achieved manually in eager mode, they may introduce tighter coupling between model code and optimization logic.</p>
<p>          <a href="https://raw.githubusercontent.com/pytorch-labs/applied-ai/refs/heads/main/assets/images/dev-discuss-asynctp/Figure_11.png" target="_blank" rel="noopener nofollow ugc" class="onebox">
            <img src="https://raw.githubusercontent.com/pytorch-labs/applied-ai/refs/heads/main/assets/images/dev-discuss-asynctp/Figure_11.png" width="690" height="302">
          </a>
<br>
<em>Figure 11: torch.compile can automatically apply async-TP to an all-gather followed by multiple matmuls that consume the all-gather result (e.g., QKV projection)</em></p>
<p>For authoring TP logic, we recommend PyTorch Tensor Parallel APIs. You can find the tutorial <a href="https://pytorch.org/tutorials/intermediate/TP_tutorial.html" rel="noopener nofollow ugc">here</a> and an example in TorchTitan <a href="https://github.com/pytorch/torchtitan/blob/1923ce4/torchtitan/parallelisms/parallelize_llama.py#L158-L183" rel="noopener nofollow ugc">here</a>. Additionally, torch.compile can apply async-TP to TP logic that is manually authored using functional collectives along with <code>torch.mm</code>, <code>torch.matmul</code>, or <code>torch._scaled_mm</code>. An example can be found <a href="https://github.com/pytorch/pytorch/blob/16b8146/test/distributed/tensor/parallel/test_micro_pipeline_tp.py#L206-L208" rel="noopener nofollow ugc">here</a>.</p>
<pre><code class="lang-auto">from torch.distributed._symmetric_memory import enable_symm_mem_for_group

# Enable symmetric memory for the TP process group
enable_symm_mem_for_group(tp_group.group_name)

# Tell torch.compile to enable async-TP
torch._inductor.config._micro_pipeline_tp = True

# Apply torch.compile to the model
model = torch.compile(model)

# Or apply torch.compile to only the model region that contains TP logic
model.tp_submodule = torch.compile(model.tp_submodule)
</code></pre>
<h4><a name="p-454286-using-async-tp-in-eager-mode-12" class="anchor" href="#p-454286-using-async-tp-in-eager-mode-12"></a>Using Async-TP in eager mode:</h4>
<p>It is also possible to apply async-TP in eager mode by directly calling async-TP ops:</p>
<pre><code class="lang-auto">from torch.distributed._symmetric_memory import enable_symm_mem_for_group

# Enable symmetric memory for the TP process group
enable_symm_mem_for_group(tp_group.group_name)

# Invoke the async-TP operators directly
# all-gather -&gt; matmul
ag_output, mm_outputs = torch.ops.symm_mem.fused_all_gather_matmul(
    x,
    [wq, wk, wv],
    gather_dim=1,
    group_name=tp_group.group_name,
)

# matmul -&gt; reduce-scatter
output = torch.ops.symm_mem.fused_matmul_reduce_scatter(
    x,
    w,
    "avg",
    scatter_dim=0,
    group_name=tp_group.group_name,
)
</code></pre>
<h2><a name="p-454286-limitations-and-future-work-13" class="anchor" href="#p-454286-limitations-and-future-work-13"></a>Limitations and Future Work</h2>
<p>Current limitations of the async-TP support in PyTorch:</p>
<ul>
<li><strong>Optimized for large matmul problems</strong>: Currently, async-TP in PyTorch performs best with large matmul operations, particularly those that don’t require changes in tiling size after decomposition. We are exploring finer-grained pipeline solutions to enhance performance on smaller problem sizes, such as those encountered in inference workloads.</li>
<li><strong>Requires NVSwitch</strong>: At present, async-TP in PyTorch relies on NVSwitch for optimal performance. We are considering extending support to NVLink ring topologies based on community feedback and demand.</li>
<li><strong>Limited to intra-node configurations</strong>: async-TP in PyTorch currently only works for intra-node setups. We may explore extending this support to cross-node environments in the future.</li>
</ul>
<h2><a name="p-454286-notes-14" class="anchor" href="#p-454286-notes-14"></a>Notes</h2>
<p><em><a name="note1"></a>[1]: PyTorch Distributed opted the term “async-TP” to describe this technique, but it may not be universally referred to by this name.</em></p>
<p><em><a name="note2"></a>[2]: Currently, the async-TP implementation in PyTorch requires an NVLink connection between all device pairs (e.g., through NVSwitch) to achieve speedup. This is a limitation that we plan to address in future.</em></p>
<p><em><a name="note3"></a>[3]: Copy engines are dedicated hardware units on a GPU that manage data movement between different memory locations and operate independently of the GPU’s computational cores (SMs).</em></p>
<p><em><a name="note4"></a>[4]: Data transfers via copy engines still share the same memory bandwidth. However, this is unlikely to cause significant contention because (1) the transfer rate is limited by NVLink bandwidth, which is low enough to avoid memory bandwidth contention, and (2) the overlapped matmul is compute-bound.</em></p>
<p><em><a name="note5"></a>[5]: We assume that the matmul problem size is large enough, so that the tile shape doesn’t change after decomposition and the main source of decomposition overhead is quantization inefficiency.</em></p>
<p><em><a name="note6"></a>[6]: The H100 GPUs used for the benchmark were non-standard. They have HBM2e and are limited to a lower TDP. The actual peak TFLOPs should be between SXM and NVL, and we don’t know the exact value. So the reported MFU is lower than the actual MFU because we use the peak TFLOPs of SXM directly.</em></p>

username: anonymous_user_83259
cooked: <p>Is there any information available about how the alternating stream symmetric memory async TP implementation compares to Flux’s implementation (<a href="https://arxiv.org/abs/2406.06858" rel="noopener nofollow ugc">https://arxiv.org/abs/2406.06858</a>)?</p>

username: jinliang_li
cooked: <p>I’d like to know that, too.</p>

username: youkaichao1
cooked: <p>second that, would like to see the comparison against the flux fused kernel.</p>

username: sky_Faded
cooked: <p>Hi, would you please check the pictures ?<br>
I think they are lost.</p>

username: Jojimon_Varghese
cooked: <p>The images in this page are not visible</p>

username: Felix-Zhenghao
cooked: <p><a class="mention" href="/u/yifuwang">@yifuwang</a> could you please check the images? They are invisible now.</p>

username: Wenlei_Bao
cooked: <p>Hi <a class="mention" href="/u/yifuwang">@yifuwang</a>,<br>
Just curious if this is still active or not? Because I saw some old PRs still open in pytorch repo. From this post it looks like first use the decomposition approach, but later improved to a more fine-grained approach (like this post) similar to <a href="https://github.com/bytedance/flux" rel="noopener nofollow ugc">Flux</a> and <a href="https://github.com/ByteDance-Seed/Triton-distributed" rel="noopener nofollow ugc">Triton-distributed</a>. So wondering if you ever perform any comparison with them?<br>
Thanks.<br>
-W</p>

username: YUNQIUGUO
cooked: <p>+1 can we check the images? They are not visible.</p>

username: Faidle
cooked: <p>Ok got it. Thank you for discussing.</p>

username: albanD
cooked: <p>Sorry about that, they are updated now!</p>

username: wenchenvincent
cooked: <p>I might have missed something but I don’t fully understand why we still need the barriers in the alternating-stream implementation shown in Figure 4. For the approach in Figure 3, synchronization is necessary because we need to wait for the data fetch from another stream to be done. However, if the data fetch (“Fetch A3” in Figure 4) and the sub-matmul that need this data (“A3@B” in Figure 4) are on the same stream, they are naturally ordered and should not require a barrier. What is the barrier after “Fetch A3” for?</p>
<p>Also, I don’t quite understand what the barrier between “Post A0” and “Fetch A1” for.</p>