Why not use multiprocessing in torchvision/reference

username: taotaotao
cooked: <p>Hi, I’m learning to write distributed program by looking at the source code of torchvision</p>
<p>I was confused when I saw this part about evaluate.</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/vision/blob/78159d61b966ec32ac330c28bea32e9e12fe6eb4/references/detection/engine.py#L103" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/vision/blob/78159d61b966ec32ac330c28bea32e9e12fe6eb4/references/detection/engine.py#L103" target="_blank" rel="noopener nofollow ugc">pytorch/vision/blob/78159d61b966ec32ac330c28bea32e9e12fe6eb4/references/detection/engine.py#L103</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="93" style="counter-reset: li-counter 92 ;">
<li>
</li><li>    res = {target["image_id"].item(): output for target, output in zip(targets, outputs)}</li>
<li>    evaluator_time = time.time()</li>
<li>    coco_evaluator.update(res)</li>
<li>    evaluator_time = time.time() - evaluator_time</li>
<li>    metric_logger.update(model_time=model_time, evaluator_time=evaluator_time)</li>
<li>
</li><li># gather the stats from all processes</li>
<li>metric_logger.synchronize_between_processes()</li>
<li>print("Averaged stats:", metric_logger)</li>
<li class="selected">coco_evaluator.synchronize_between_processes()</li>
<li>
</li><li># accumulate predictions from all images</li>
<li>coco_evaluator.accumulate()</li>
<li>coco_evaluator.summarize()</li>
<li>torch.set_num_threads(n_threads)</li>
<li>return coco_evaluator</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p><strong>coco_evaluator</strong> is a warpper for  <strong>pycocotools.cocoeval.COCOeval</strong></p>
<p>When all the processes have been evaluated, we need to gather all the results.</p>
<p>The core of <strong>synchronize_between_processes</strong> is <strong>torch.distributed.all_gather</strong></p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/vision/blob/78159d61b966ec32ac330c28bea32e9e12fe6eb4/references/detection/utils.py#L75" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/vision/blob/78159d61b966ec32ac330c28bea32e9e12fe6eb4/references/detection/utils.py#L75" target="_blank" rel="noopener nofollow ugc">pytorch/vision/blob/78159d61b966ec32ac330c28bea32e9e12fe6eb4/references/detection/utils.py#L75</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="65" style="counter-reset: li-counter 64 ;">
<li>
</li><li>    def __str__(self):</li>
<li>        return self.fmt.format(</li>
<li>            median=self.median,</li>
<li>            avg=self.avg,</li>
<li>            global_avg=self.global_avg,</li>
<li>            max=self.max,</li>
<li>            value=self.value)</li>
<li>
</li><li>
</li><li class="selected">def all_gather(data):</li>
<li>    """</li>
<li>    Run all_gather on arbitrary picklable data (not necessarily tensors)</li>
<li>    Args:</li>
<li>        data: any picklable object</li>
<li>    Returns:</li>
<li>        list[data]: list of data gathered from each rank</li>
<li>    """</li>
<li>    world_size = get_world_size()</li>
<li>    if world_size == 1:</li>
<li>        return [data]</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>That’s follow those steps:</p>
<ol>
<li>coco.Eval computed on CPU in single process</li>
<li>
<strong>synchronize_between_processes</strong> transfer coco evaluate result to GPU</li>
<li>doing all_gather over all process on GPU</li>
<li>all_gather result transfer to CPU for logger</li>
</ol>
<p>I’s so trouble, Why not just use CPU  directly, here are some other ways，like, <strong>pipe</strong> or <strong>queue</strong> communicate between processes.</p>
<p>Is it because there is some reason here, to use GPU instead of  <strong>multiprocessing.pipe/multiprocessing.queue</strong>?</p>