Federated Learning without distributing the same data

abodh_ltd: <p>Hello All,</p>
<p>I am trying to implement federated learning for my research. However, the tutorials or code snippets that I found (on MNIST) distribute a collected MNIST dataset over n number of clients.</p>
<p>To make this simple, suppose I have just 2 clients with different data as inputs(for eg. client A has data 1-5 and client B has data 6-10). I want to train the model separately on Client A and Client B with their respective data (not distributing the combined dataset). Is this possible to do so? can you please give me an example to proceed further?</p>
<p>To clarify on my current setup, I am using Windows 10 with pytorch 1.0.0 and do not know if additional libraries like pysyft works for my version. I just want to make it work using pytorch.</p>
<p>I would really appreciate anybody’s help on this.</p>

mrshenli: <p>I assume Client A and Client B need to communicate (maybe through a master) at some point to average the model? What communication pattern (e.g., collective vs RPC) will they be using?</p>
<p>We do have <a href="https://pytorch.org/docs/stable/distributed.html" rel="nofollow noopener">collective</a> and <a href="https://pytorch.org/docs/stable/rpc.html" rel="nofollow noopener">RPC</a> API, but they are not yet available on Windows. If you need this on Windows, could you please comment on <a href="https://github.com/pytorch/pytorch/issues/37068" rel="nofollow noopener">this poll</a> to help us better prioritize it?</p>

abodh_ltd: <p>Yes it’s just like in federated learning. client A and client B will be training their own data for respective number of epochs E. Then they need to send the gradients to a master who will average the model and send it back to the clients. I do have some access to Linux system. I would appreciate if you could provide me an example to do this.</p>
<p>P.S. Can you tell how to make sure that each of the model has different input? currently all of the examples use a same dataloader to get the input samples and just randomly distribute the data as input. I need to have each of the model train separately on their own data.</p>

mrshenli: <p>Since this is master/client training scenario, it might fit into the parameter server paradigm. Please see <a href="https://pytorch.org/tutorials/intermediate/rpc_param_server_tutorial.html" rel="noopener nofollow ugc">this tutorial</a> and <a href="https://github.com/pytorch/examples/tree/master/distributed/rpc/parameter_server" rel="noopener nofollow ugc">this example</a> for a complete example.</p>
<blockquote>
<p>Can you tell how to make sure that each of the model has different input?</p>
</blockquote>
<p>In this case, as each client is running in its own process, it can create its dedicated dataloader. Or if you need the master to dispatch the input data to those clients, this is also doable. You can wrap those client logic into a function and let the master to launch them, sth like  <code>rpc.rpc_async("client_a", client_func, args=(inputs_for_a,))</code>.</p>
<blockquote>
<p>I need to have each of the model train separately on their own data.</p>
</blockquote>
<p>Since the forward-backward does not need to span across machine boundaries, <a href="https://pytorch.org/tutorials/intermediate/rpc_tutorial.html#distributed-reinforcement-learning-using-rpc-and-rref" rel="noopener nofollow ugc">this RPC+RRef Example</a> might be sufficient, as you don’t need distributed autograd and distributed optimizer.</p>