Active learning and distributed data parallel

username: abhay
cooked: <p>Hi,</p>
<p>I recently started looking into using pytorch for an active learning project and have some queries in using distributed data parallel. As per my understanding, distributed data parallel creates a process per gpu to run the model. Active learning approaches usually have two stages: annotation and training. In the annotation stage, I need to run the model on the unannotated samples and get annotations for selected samples and in the training stage, the model needs to be trained on the annotated dataset.  I am trying to use DistributedDataParallel but am not sure how to.</p>
<p>My workflow will roughly look like:</p>
<ol>
<li>Run the model on unannotated samples (across all gpus)</li>
<li>rank samples using the output from previous step based on some criterion (main worker, rank 0)</li>
<li>add new samples to the training dataset (all workers’ data loaders need to be updated) and then run training.</li>
</ol>
<p>This cycle will repeat till a budget is hit (that budget needs to be synced across all processes in order to be able to terminate them).</p>
<p>I am not clear on how I get the output of the model from all processes (step 1) back into the main process, run things like the sample ranking (step 2) on the main process while the other processes wait for the input from the main process before starting the training cycle again. Also, how do I sync variables like budget across processes?  It would be great if I can get a pointer to a resource which helps me understand this more.  I have looked at the imagenet training script in the github repo but that didn’t help me understand this process.</p>
<p>Thanks!</p>

username: abhay
cooked: <p>I have been looking into the distributed operations defined <a href="https://pytorch.org/docs/stable/distributed.html" rel="nofollow noopener">here</a>. Are operations like gather and reduce blocking? For example, after running the model on unlabeled data, I can call gather to get them on a single host. If I do that, do the other processes get blocked as well. The main process then needs to select samples for the labeled set and then broadcast them to all the processes. How do I block the other processes to receive this broadcast before beginning the training cycle?</p>

username: iffiX
cooked: <p>The collective communications are designed for tightly coupled processes, to give you a better idea of how they work, I borrowed some images from mpi:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/6/3/638f9f193a3998a9d1251ce140adebb277743a05.png" alt="image" data-base62-sha1="ecKYAfMKAvBdma2Jr47g32U0GP3" width="280" height="154"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/1/8/18e87c687f2ad849b761c0e68e74375da9f0491d.png" alt="image" data-base62-sha1="3ylyBEGFhgRL87KpxBY0OOF1WyN" width="287" height="340"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/6/b/6bcc27f3ec011808d576480906a657b51c40e156.png" alt="image" data-base62-sha1="fnCuu3RnN96pGC1oebWT4FLHEUK" width="211" height="169"><br>
Each collective communication primitive is a <strong>blocking</strong> process for <strong>all processes</strong> in your created processgroup (whether it is the global WORLD group or a sub group of the global group).</p>
<p>I would suggest you do it in the following way:</p>
<pre><code class="lang-auto">import torch.distributed as dist

# make sure all processes in your_group will run this step
model = (model, ..., output_device=your_device, process_group=your_group)
your input per process= ...

# output is only visible to that process
# parameters are synchronized behind the stage
output = model(your_input) 

# make sure all processes in your_group will run this step
dist.gather(..., dst=0)
# perform ranking on process 0
dist.scatter(..., src=0)

# all workers are now synchronized
</code></pre>
<p>One <strong>crutial</strong> thing to notice: the receive buffer (tensor) must be equal or larger than the tensor you have sent, otherwise nasty errors will be thrown.</p>

username: abhay
cooked: <p>Thank you very much for the images! I was confused with some of the collective operations and that image really helped clear things up.</p>