akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
What is the uplimit of dynamo graph cache?

username: GodHforever
cooked: <p>I am curious about the uplimit of dynamo graph cache.  If there are too many graph breaks or too many kinds of guards for full graph, will dynamo save all compiled graph for users?</p>
<p>On the other hand, if dynamo will discard some graphs by order when cache is full, does that means we always need to recompile the model in some conditions?</p>

username: GodHforever
cooked: <p>I found docs in <a href="https://pytorch.org/docs/2.6/torch.compiler_troubleshooting.html#changing-the-cache-size-limit" class="inline-onebox" rel="noopener nofollow ugc">torch.compile Troubleshooting — PyTorch 2.6 documentation</a>. However just limit the cache size may not be a good solution. Is it possible to save caches in custom backend?</p>

username: bdhirsh
cooked: <p>You can bump “torch._dynamo.recompile_limit” and “torch._dynamo.accumulated_recompile_limit” to a higher value if you want compile to support storing a larger number of compiled graphs for a given frame.</p>
<p>Although generally the reason that those defaults are set is that:</p>
<p>(1)  have an extremely large number of recompiles, you will probably suffer from pretty long compile times</p>
<p>(2) a large number of recompiles can often indicate that there is some code in your model that we might be (unnecessarily) specializing on, that you can fix with some model tweaks.</p>
<p>Do you know what’s causing all of the recompiles you’re seeing? You can diagnose by running with “TORCH_LOGS=recompiles” and dumping the output</p>
Different number of GPUs with DDP give different results

username: lbadi
cooked: <p>Consider following cases:</p>
<ol>
<li>I train my network with 2 GPUs such that batch size = dataset size/2.</li>
<li>I train my network with 4 GPUs such that batch size = dataset size/4.<br>
Assume dataset size is divisible by 4.</li>
</ol>
<p>Results I get in each case is different. I fix all random seeds.</p>

username: ptrblck
cooked: <p>This is expected as PyTorch does not guarantee to create bitwise identical and deterministic results in different setups.</p>

username: lbadi
cooked: <p>Maybe I might not have made myself clear. Let n1 be number of gpus in case 1 and n2 be number of gpus in case 2. In case 1, batch size per gpu is (dataset size)/n1 i.e. total batch size is dataset size. Similarly for case 2. Gist is that total batch size/effective batch size is same in both cases. But results in each case are different. Are you saying the results will be different in these 2 cases? If so, how can I make to be for varying number of gpus?</p>
<p>Thanks for your reply.</p>

username: ptrblck
cooked: <p>Yes, since the local batch size differs between your setups, different algorithms can be selected in math libraries. This could then not create bitwise identical outputs between the setups, but should still be deterministic if the same setup is executed repeatedly.</p>
How Does PyTorch’s FSDP Handle Gradients for Unsharded Parameters During Backward Pass?

username: erfan_mhi
cooked: <p>I was reviewing PyTorch’s Fully Sharded Data Parallel (FSDP) implementation and its associated paper but came across a point of confusion. It seems that after the forward pass, all unsharded parameters are discarded. However, my understanding is that we need to retain their activations to compute gradients for the unsharded parameters during the backward pass.</p>
<p>Are the activations for unsharded parameters still being kept somewhere? If not, how does FSDP compute gradients for unsharded parameters during backward pass on the same batch of data available on the current device?</p>
Kernel density estimation in Pytorch

username: cbd
cooked: <p>Is kernel density estimation function available in pytorch?</p>
<p>If not i need to use detach during conversion from tensor to numpy for kernel density estimation function but it want back propagate gradient. Is there any workaround solution?</p>

username: googlebot
cooked: <aside class="quote no-group" data-username="cbd" data-post="1" data-topic="110157">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/13edae/48.png" class="avatar"> cbd:</div>
<blockquote>
<p>Is kernel density estimation function available in pytorch?</p>
</blockquote>
</aside>
<p>No. You will usually need more specialized routines depending on what is trained (knots, bandwidth, density of observations). Mixture density networks or cubic splines could work as “workarounds”, but these are not bundled either of course.</p>

username: kkladny
cooked: <p>I created a proper PyTorch package called <a href="https://github.com/rudolfwilliam/torch-kde" rel="noopener nofollow ugc">torch-kde</a> of the <a href="https://scikit-learn.org/1.6/modules/generated/sklearn.neighbors.KernelDensity.html" rel="noopener nofollow ugc">KernelDensity class from scikit-learn</a> through which you can backpropagate. Enjoy!</p>
Executorch Vulkan delegate on embedded Linux

username: Alex_G_Gener
cooked: <p>Hello. I’m using a embedded linux platform and want to deploy pytorch models on the ARM MALI GPU. I’m setting the Vulkan delegate but having problems when building the runner with Vulkan. All tutorials I’ve found are related to Android development and Vulkan linking of libraries is done automatically with the Android NDK, but I’m not able to link the CMAKE to my GLSL compiler. I have glslangValidator installed instead of glslc.</p>
<p>Is that supported? Are there any other limitations we should consider when using the Vulkan delegate out of Android?</p>
<p>Thanks in advance,<br>
Alex.</p>

username: kimishpatel
cooked: <p><a class="mention" href="/u/ssjia">@ssjia</a> can you answer this one?</p>

username: ssjia
cooked: <p>Hello <a class="mention" href="/u/alex_g_gener">@Alex_G_Gener</a>, thanks for your interest in the Vulkan delegate and apologies for the super late reply. Anecdotally, I am able to compile and execute the Vulkan delegate on a Linux platform. You can try the same with the following commands</p>
<pre data-code-wrap="bash"><code class="lang-bash">cd &lt;executorch_root&gt;

# Clean repo
./install_requirements.sh --clean
git submodule sync
git submodule update --init

# Install requirements
./install_requirements.sh --pybind

# Build ExecuTorch libs with Vulkan
(rm -rf cmake-out &amp;&amp; \
  cmake . \
  -DCMAKE_INSTALL_PREFIX=cmake-out \
  -DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON \
  -DEXECUTORCH_BUILD_EXTENSION_MODULE=ON \
  -DEXECUTORCH_BUILD_EXTENSION_RUNNER_UTIL=ON \
  -DEXECUTORCH_BUILD_EXTENSION_TENSOR=ON \
  -DEXECUTORCH_BUILD_KERNELS_OPTIMIZED=ON \
  -DEXECUTORCH_BUILD_KERNELS_CUSTOM=ON \
  -DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \
  -DEXECUTORCH_BUILD_TESTS=ON \
  -DEXECUTORCH_BUILD_VULKAN=ON \
  -Bcmake-out &amp;&amp; \
  cmake --build cmake-out -j64 --target install)

# Build Vulkan test binary
(rm -rf cmake-out/backends/vulkan/test &amp;&amp; \
  cmake backends/vulkan/test \
    -DCMAKE_INSTALL_PREFIX=cmake-out \
    -DPYTHON_EXECUTABLE=python \
    -Bcmake-out/backends/vulkan/test &amp;&amp; \
  cmake --build cmake-out/backends/vulkan/test -j16)

# Execute test binary
cmake-out/backends/vulkan/test/vulkan_compute_api_test --gtest_filter="*"
</code></pre>
<p>As for <code>glslc</code> vs <code>glslangValidator</code>, there is currently an assumption that <code>glslc</code> is the shader compiler of choice since that is what’s included in the Android NDK. However, it should be easy to add a build configuration to specify that <code>glslangValidator</code> should be used instead.</p>
Get a non-leaf tensor grad to its output

username: spencer03172023
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/2/d27ee8319cdfa52a67befa4c6b19332a08387aa9.png" data-download-href="https://discuss.pytorch.org/uploads/default/d27ee8319cdfa52a67befa4c6b19332a08387aa9" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d27ee8319cdfa52a67befa4c6b19332a08387aa9_2_690x371.png" alt="image" data-base62-sha1="u28ahltcNP8rHf1O6HWvwAr8doR" width="690" height="371" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d27ee8319cdfa52a67befa4c6b19332a08387aa9_2_690x371.png, https://discuss.pytorch.org/uploads/default/original/3X/d/2/d27ee8319cdfa52a67befa4c6b19332a08387aa9.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/2/d27ee8319cdfa52a67befa4c6b19332a08387aa9.png 2x" data-dominant-color="F2F5FA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">954×514 12.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I unroll a nerual network as the picture. Then i want to have the grad value of each CNN output to its input, for example:<br>
below is my partical code:</p>
<h1><a name="p-462930-compute-gradient-1" class="anchor" href="#p-462930-compute-gradient-1"></a>Compute gradient</h1>
<pre><code>    x.requires_grad_(True)
    D_x = self.denoise_operator(x)
    g = x - D_x - z + u
    gradient = ATWAx_b + self.rho_param * g
    
    # Backward pass to compute ∇x D(x)
    D_x.backward(torch.ones_like(D_x), retain_graph=True)
    D_x_grad = x.grad.clone()
    x.grad.zero_()  # Clear the gradient to avoid accumulation
</code></pre>
<p>this can work in the 1st iteration of the unrolling NN, but x.grad is None type error in the 2nd iteration of the NN, as mentioned before, the second partial derivative can’t work.</p>
<p>Any one can help? Thanks</p>

username: soulitzer
cooked: <p>Are you looking for something like <a href="https://pytorch.org/docs/stable/generated/torch.Tensor.retain_grad.html" class="inline-onebox" rel="noopener nofollow ugc">torch.Tensor.retain_grad — PyTorch 2.5 documentation</a></p>
IndexError: pop from empty list2

username: JeffffFu
cooked: <p>I want to add opacus to my model, but when I start the second round of training, the following error occurs: (there is no problem with only one training session)</p>
<pre><code class="lang-auto">loss.backward()
</code></pre>
<p>File “/mnt/DataDisk/conda/envs/syft/lib/python3.7/site-packages/torch/tensor.py”, line 245, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)<br>
File “/mnt/DataDisk/conda/envs/syft/lib/python3.7/site-packages/torch/autograd/<strong>init</strong>.py”, line 147, in backward<br>
allow_unreachable=True, accumulate_grad=True) # allow_unreachable flag<br>
File “/mnt/DataDisk/xierong/opacus/opacus/grad_sample/grad_sample_module.py”, line 199, in capture_backprops_hook<br>
module, backprops, loss_reduction, batch_first<br>
File “/mnt/DataDisk/xierong/opacus/opacus/grad_sample/grad_sample_module.py”, line 237, in rearrange_grad_samples<br>
A = module.activations.pop()<br>
IndexError: pop from empty list</p>
<p>I have no idea, can you help give some advice?</p>
<p>My code as follows:<br>
def train_model(self, train_loader, test_loader, save_name=None):<br>
self.model = self.model.to(self.device)<br>
optimizer = optim.Adam(self.model.parameters(), lr=self.args[‘lr’])<br>
if self.args[‘optim’] == “SGD”:<br>
optimizer = optim.SGD(self.model.parameters(), lr=self.args[‘lr’], momentum=0)</p>
<pre><code>    if self.args['is_dp_defense']:
        privacy_engine = PrivacyEngine(secure_mode=False)

        self.model, optimizer, train_loader = privacy_engine.make_private(
            module=self.model,
            optimizer=optimizer,
            data_loader=train_loader,
            noise_multiplier=0.5,
            max_grad_norm=1.0,
        )


    criterion = nn.CrossEntropyLoss()
    losses = []
    run_result = []

    self.model.train()

    for epoch in range(self.args['num_epochs']):
        losses = []
        for data, target  in train_loader:

            data, target = data.to(self.device), target.to(self.device)
            optimizer.zero_grad()
            output = self.model(data)
            loss = criterion(output, target)
            loss.backward()
            optimizer.step()
            losses.append(loss.item())

        if self.args['is_dp_defense']:
            epsilon = privacy_engine.accountant.get_epsilon(delta=1e-5)
            print(
                f"Train Epoch: {epoch} \t"
                f"Loss: {np.mean(losses):.6f} "
                f"(ε = {epsilon:.2f}, δ = 1e-5)"
            )
        # else:
        #     print(f"Train Epoch: {epoch} \t Loss: {np.mean(losses):.6f}")

        train_acc = self.test_model_acc(train_loader)
        test_acc = self.test_model_acc(test_loader)
        print('epoch %s: train acc %s | test acc %s ' % (epoch,  round(train_acc, 4),  round(test_acc, 4)))
        # self.logger.debug('epoch %s: train acc %s | test acc %s | ovf %s' % (epoch, train_acc, test_acc, train_acc - test_acc))
        run_result.append([epoch, np.mean(losses), train_acc, test_acc])
</code></pre>
Sm_90a support?

username: Boltzmachine
cooked: <p>I encounter this error when I install <code>torch_geometric</code> but I think it is likely an error of pytorch</p>
<pre><code class="lang-auto">File "/home/ubuntu/miniconda3/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1972, in &lt;listcomp&gt;
          supported_sm = [int(arch.split('_')[1])
      ValueError: invalid literal for int() with base 10: '90a
</code></pre>
<p>I found this is because</p>
<pre><code class="lang-auto">torch.cuda.get_arch_list()
&gt;&gt;&gt; ['sm_50', 'sm_80', 'sm_86', 'sm_89', 'sm_90', 'sm_90a']
</code></pre>
<p>looks like <code>sm_90a</code> triggers this error.</p>

username: ptrblck
cooked: <p>Which PyTorch version are you using? I assumed the latest release already fixed this issue.</p>

username: Boltzmachine
cooked: <p>I use 2.5.1, which is the latest version?</p>

username: ptrblck
cooked: <p>Related issue: <a href="https://github.com/pytorch/pytorch/issues/144037" class="inline-onebox">cpp_extension.py expects an integer on CUDA_ARCH, failing with Grace Hopper. · Issue #144037 · pytorch/pytorch · GitHub</a> with my follow up.</p>
How to Use Custom fp8 to fp16 Datatype Represented in uint8 in PyTorch

username: erfan_mhi
cooked: <p>Hi everyone,</p>
<p>I’ve been working on a project where I wrote a custom datatype conversion from fp16 to fp8 (E4M3 and E5M2), which is represented in <code>uint8</code> in PyTorch. I’m a bit stuck on how to use this datatype properly in my PyTorch code. I want to add this new data type to PyTorch so I can use it for mixed precision training later.</p>
<p>Could someone guide me on the best practices for implementing and utilizing this custom datatype in PyTorch, considering that it is represented in <code>uint8</code>?</p>

username: ptrblck
cooked: <p><code>torchao</code>’s <code>Float8Tensor</code> defined <a href="https://github.com/pytorch/ao/blob/457c5b1400b61f1a6141183ff02b2d1528f637e1/torchao/float8/float8_tensor.py#L242">here</a> could be a good starter for your custom tensor class.</p>
Best Practices for Running FSDP: Kubernetes, Ray, or Slurm?

username: erfan_mhi
cooked: <p>For managing <code>torchrun</code> across nodes and ensuring fault tolerance in Fully Sharded Data Parallel (FSDP) training, which would be the best option:</p>
<ol>
<li><strong>Kubernetes</strong></li>
<li><strong>Ray</strong></li>
<li><strong>Slurm</strong></li>
</ol>
<p>I’m looking for the approach that aligns best with current industry practices for scalability, fault tolerance, and speed in distributed deep learning. For instance, in terms of speed, I’ve read that Kubernetes can add ~2% overhead due to containerization but Slurm doesn’t have this issue.</p>
Dual CPU socket RAM utilization (Why is only one socket being used?!)

username: taehyunzzz
cooked: <p>Hello, I was wondering if anyone has experience looking into the RAM memory bandwith when running Pytorch CPU - whether memory bandwidth from both sockets are being used.</p>
<p>I have a setting with 8 DRAMs, 4 for each CPU socket (NUMA).<br>
When I use intel PCM memory profiling to see the bandwidth utilization while running my Pytorch CPU code for large matmuls, I found that only 1 socket was being used.</p>
<p><strong><img src="https://lh7-rt.googleusercontent.com/slidesz/AGV_vUd2B89VtN-BnMz9wEpT6QEHTct438JMrde0eR5OXK9R5fBuXK229uT59gsi2zNbsyt4UPdTlNCSNeqni9zMOAMmGlcbGjCQGXtbWroboByNOhEbF4j_fQ_SuOYxpPu6qBXm9GX4Yw=s2048?key=-sd5fIIXCCCvRNpf5Vtz6sQJ" alt="|960px;x304px;" width="690" height="218"></strong></p>
<p>I saw some posts that say this was deliberately done to avoid remote NUMA accesses.<br>
But I want a workaround BAD, because I think CPU performance could actually be a lot better <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>
<p>Any comments? Would be grateful for any workarounds <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"><br>
You could suggest other frameworks (e.g., llama.cpp) or optimizations (torchcpu, torch.compile) that may make this happen!</p>
<p>Taehyun</p>
Pytorch meets deadlock when loading nn.Module with lightning.Fabric

username: forestbat
cooked: <p>When I try to use <code>lightning.Fabric.setup()</code> to load <code>torch.nn.Module</code> under multi-process, the program will meet deadlock and stuck in <code>lightning.fabric.strategies.launchers.subprocess_script</code>.</p>
<p>I doubt this problem comes from <code>popen</code> start method of process, but I have not more evidence.</p>
<p>I tried to reproduce it in <a href="https://pastebin.com/XYLqN3Xd" rel="noopener nofollow ugc">this demo</a>, but I failed, so it’s a pity that I can’t provide minimal reproducible example now.</p>
<p>And I have seen <a href="https://discuss.pytorch.org/t/ddp-via-lightning-fabric-training-hang-with-100-gpu-utilization/181046">this topic</a> in the forum, but it can’t help me solve the problem.</p>
<p>Is there any other method to solve the problem or reproduce the problem and get enough information to debug at least?</p>
<p>Hope your reply.</p>
The scope of `torch.no_grad` and `torch.inference_mode`

username: AviZ
cooked: <p>How deep is the scope of <code>torch.no_grad</code> and <code>torch.inference_mode</code>?</p>
<p>At the following code:</p>
<pre data-code-wrap="python"><code class="lang-python">with torch.no_grad():
    y = my_fun(my_tensor)
</code></pre>
<p>Will it affect <code>my_fun</code>? Or must I add <code>@torch.no_grad</code> in its definition?<br>
What if tensors are generated within <code>my_fun</code>?</p>
<p>I assume the answer <code>torch.inference_mode</code> will hold for as well.</p>

username: ptrblck
cooked: <p>The context is applied to all methods invoked inside it unless a new context manager is used (at a nested level) changing the behavior.</p>
How to combine pre-trained weights of components from different multimodal LLMs?

username: tcm
cooked: <p>Hi everyone,</p>
<p>I’m currently doing some research on multimodal LLMs and as you know, an MLLM has multiple models from vision, text, and speech, etc. in it. Most MLLMs have vision/ audio encoder(s) to extract features from images, videos, and audio, and some connection modules to adapt the features to the embedding space of the LLM, and usually only the connection modules are trained, with the pre-trained encoders and LLM frozen.</p>
<p>I’m trying to combine some components (e.g. vision encoders) from an MLLM with the architecture of another MLLM, and these MLLMs have their weights stored in safetensors files of Hugging face repos. So my question is, do we have a way to inspect those safetensors files to know which sets of weights correspond to which components in the MLLM? And a possibly more difficult question is that can we combine parts of the weights from this MLLM’s safetensors (e.g. only the vision encoder’s weights) to the weights in safetensors of the other MLLM?</p>
<p>Thanks in advance.</p>
How to Implement Flash Attention in a Pre-Trained BERT Model on custom dataset?

username: Karam1533
cooked: <p>I have a fully pre-trained model on my custom dataset. In its current implementation, flash attention is not used, so the model is running without it. I want to enable flash attention in the model.</p>
<p>I have two specific questions:</p>
<ol>
<li>If I modify the code by simply <code>torch.nn.functional.scaled_dot_product_attention</code> flash attention implementation, will that enable flash attention correctly?</li>
<li>After making this modification, do I need to finetune the model, or can I use it as-is without further training?</li>
</ol>
<p>I would greatly appreciate any clarification or guidance on this! Thanks in advance.</p>
Torchvision ResNet Input Size

username: Finspire13
cooked: <p>Hi, I am playing with the pre-trained Resnet101 in torchvision. I tried different input size of images (224x224, 336x336, 224x336) and it seem all works well. So what’s the exact valid range of input size to send into the pre-trained ResNet?</p>

username: Md_Mahfujur_Rahman_0
cooked: <p>I think the valid input size of images is 224<em>224. May be you are using preprocessing in your code and whatever the input size of the image is, it crop as just 224</em>224. Thanks.</p>

username: Finspire13
cooked: <p>I printed modules in the ResNet and found why:</p>
<p>The AvgPool before the last FC is like this:<br>
(avgpool): AvgPool2d (size=7, stride=7, padding=0, ceil_mode=False, count_include_pad=True)</p>
<p>Therefore as long as the input image size makes the AvgPool output tensors of size 1x2048x1x1, there is no problem. But if the input size is not 224x224, it is cropped by ResNet implicitly at AvgPool layer.</p>

username: n0obcoder
cooked: <aside class="quote no-group" data-username="Finspire13" data-post="1" data-topic="9405">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/finspire13/48/1685_2.png" class="avatar"> Finspire13:</div>
<blockquote>
<p>hvision. I tried different input size of images (224x224</p>
</blockquote>
</aside>
<p>No the resnet18 model architecture itself has AdaptiveAvgPool2d layer at the end of it. This layer ensures that any size of the input image get converted to a fixed size output, so that varying input sizes would not be a problem for the dense layers.</p>

username: GeorgeIrwin
cooked: <p><a class="mention" href="/u/n0obcoder">@n0obcoder</a> , Is there a variant of resnet which is optimized to any image size? Without pretrained weights, just the builder class.</p>

username: MUSTAFA_CAGRI_CIVICI
cooked: <p>What i understand is that resnet18 has been developed with 224x244 format. The pretrained model especially for Convolution Layer or it was optimized by 224x224. If you want to train the model more with any input, the shape of input would not has any issues about the shape but the model performance would not give as much as it has. it is what I understood from some examples. If i am wrong, please make me correct <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Question: Overlapping AllGather and ReduceScatter in FSDP Backward for Better Communication Performance

username: nariaki3551
cooked: <h2><a name="p-462846-abstract-1" class="anchor" href="#p-462846-abstract-1"></a>Abstract</h2>
<p>We are exploring the idea of overlapping AllGather and ReduceScatter in FSDP’s backward pass. This approach can reduce total communication time by up to half, especially in environments that support in-network computing (e.g., NVIDIA SHARP). It could significantly speed up FSDP workflows where communication is a bottleneck.</p>
<p>The basic concept is also discussed in <a href="https://sc24.supercomputing.org/proceedings/tech_paper/tech_paper_pages/pap545.html" rel="noopener nofollow ugc">Network-Offloaded Bandwidth-Optimal Broadcast and Allgather for Distributed AI</a>, presented at SC24.</p>
<h2><a name="p-462846-background-2" class="anchor" href="#p-462846-background-2"></a>Background</h2>
<p>During the FSDP backward pass, the following collectives typically run in sequence:</p>
<ol>
<li><strong>AllGather</strong> for parameter unsharding</li>
<li><strong>ReduceScatter</strong> for gradient aggregation</li>
</ol>
<p>By using in-network computing or multicast for these operations, it’s possible to reduce either the amount of data received or sent compared to ring-based algorithms:</p>
<ol>
<li><strong>ReduceScatter with in-network computing:</strong> Decreases the amount of data each rank receives, while keeping the amount sent the same
<ul>
<li>e.g. NVIDIA SHARP, <a href="https://cloud.google.com/blog/topics/developers-practitioners/optimize-training-performance-reduction-server-vertex-ai?hl=en" rel="noopener nofollow ugc">Reduction Server</a></li>
</ul>
</li>
<li><strong>AllGather with multicast:</strong> Decreases the amount of data each rank sends, while keeping the amount received the same
<ul>
<li>e.g. NVIDIA SHARP, MVAPICH MPI</li>
</ul>
</li>
</ol>
<p>If we execute these collectives in an overlapping manner, the data receiving and sending times of the two collectives are equalized, and as a result, the overall communication time is shortened.</p>
<p>In-network computing solutions (e.g., SHARP) have become widely available and increasingly cost-effective, motivating further exploration of this approach.</p>
<h2><a name="p-462846-questions-3" class="anchor" href="#p-462846-questions-3"></a>Questions</h2>
<ol>
<li>
<p><strong>Technical feasibility</strong><br>
Are there any major obstacles to running these collective communications in parallel? For example, do we need separate CUDA streams or non-blocking collective calls to make this work?</p>
</li>
<li>
<p><strong>Performance considerations</strong><br>
Do you think this overlapping strategy would yield substantial benefits in practice, or would overhead (e.g., synchronization or stream contention) limit the gains?</p>
</li>
</ol>
<p>Any insights, shared experiences, or best practices would be greatly appreciated. Thank you.</p>