BatchNorm for multi GPU Training

username: PhilipMay
cooked: <p>Hi,<br>
I did read that PyTorch is not supporting the so called sync BatchNorm. This is needed to<br>
train on multi GPU machines. By question is: Are there any plans to implement sync BatchNorm<br>
for PyTorch and when will it be released?</p>
<p>An other question: What is the best workaround when you want to train with images and need<br>
large batch sizes?</p>
<p>Thanks<br>
Philip</p>

username: ptrblck
cooked: <p><a href="https://pytorch.org/docs/stable/nn.html?highlight=syncbatchnorm#torch.nn.SyncBatchNorm">SyncBatchNorm</a> is already in PyTorch.</p>

username: PhilipMay
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> ,<br>
thanks for the answer.<br>
The documentation sais:</p>
<blockquote>
<p>Currently SyncBatchNorm only supports DistributedDataParallel with single GPU per process.</p>
</blockquote>
<p>This “single GPU” kind of irritates me. What does it mean?</p>
<p>I am also asking because detectron2 still uses “FrozenBatchNorm2d”: <a href="https://github.com/facebookresearch/detectron2/blob/master/detectron2/modeling/backbone/resnet.py#L50" rel="noopener nofollow ugc">https://github.com/facebookresearch/detectron2/blob/master/detectron2/modeling/backbone/resnet.py#L50</a></p>

username: ptrblck
cooked: <p><code>DistributedDataParallel</code> can be used in two different setups as given in the <a href="https://pytorch.org/docs/master/nn.html#torch.nn.parallel.DistributedDataParallel">docs</a>.</p>
<ol>
<li>Single-Process Multi-GPU and</li>
<li>Multi-Process Single-GPU, which is the fastest and recommended way.</li>
</ol>
<p><code>SyncBatchNorm</code> will only work in the second approach.</p>
<p>I’m not sure, if you would need <code>SyncBatchNorm</code>, since <code>FrozenBatchNorm</code> seems to fix all buffers:</p>
<blockquote>
<p>BatchNorm2d where the batch statistics and the affine parameters are fixed.<br>
It contains non-trainable buffers called<br>
“weight” and “bias”, “running_mean”, “running_var”,<br>
initialized to perform identity transformation.</p>
</blockquote>

username: rvandeghen
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>How do I create my DDP model if I’m working on a cluster with multiple nodes and each node may have multiple GPUs ?</p>

username: ptrblck
cooked: <p>I think <a href="https://pytorch.org/tutorials/intermediate/dist_tuto.html">this tutorial</a> might be a good introduction to the different backends etc.</p>

username: pading
cooked: <p>hi, i want to know the difference between sycBN and BN.</p>

username: ptrblck
cooked: <p><code>SyncBatchNorm</code> synchronizes the statistics during training in a <code>DistributedDataParallel</code> setup as given in the <a href="https://pytorch.org/docs/stable/generated/torch.nn.SyncBatchNorm.html?highlight=syncbatchnorm#torch.nn.SyncBatchNorm">docs</a> and can optionally be used.</p>

username: acmilannesta
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> , so SyncBatchNorm won’t work on a Ray cluster of 2 workers with each worker has &gt; 1 gpu?</p>

username: ptrblck
cooked: <p>Sorry, I’m not familiar with <code>Ray</code> and don’t know why it shouldn’t work (assuming the actual use case is similar or equivalent to <code>DDP</code>).</p>