Parallel / distributed training

A set of classes to synchronize SGD between multiple solvers. Based on the Hogwild paper, and our work at Flickr to extend the model to GPUs and distributed configurations by streaming gradients between solvers.

## Features

* Models can be trained in parallel without modification. Caffe’s training code is also mostly untouched.
* Modular design. The code is broken down in simple components that synchronize one segment, e.g. CPU/GPU, CPU/LAN. They can be combined to create an architecture, either in-process or between processes by memory-mapping the weights to /dev/shm.
* Works on commodity hardware. Apparently even on 1G Ethernet, at least for mnist. Synchronization and SGD run asynchronously, to keep both compute and networking resources fully utilized. Bandwidth and latency across machines are optimized using raw sockets and user-space networking.
* No additional memory used on the GPUs.

## Limitations

* Only supports data-parallelism. Limited forms of model-parallelism should be possible using the same components but no work has been done.
* Training is less stable than on a single GPU. In particular, disabling momentum at least at the beginning of training seems to help.
* No deployment / monitoring tools. We are looking at integrating with IPython.parallel.

## Tests

Early results on MNIST seem to show linear scaling. We tested on up to 6 machines with 4 solvers each for CPU, and 2 machines with 2 GPUs each. GPUs do not perform well on this small network but still seem to scale linearly.

![mnist](https://cloud.githubusercontent.com/assets/921950/4366860/9187ad08-42c7-11e4-8734-73721efe129f.png)

In the weeks to come we plan to start testing on larger networks and clusters. Currently our GPU machines are connected through 1G Ethernet, please contact us if you are interested to help benchmarking on better hardware.

## Architecture

We made the Caffe singleton thread-local, to allow multiple solvers to run in parallel on their own thread. Synchronization works by sharing the weight buffers between solvers in the same address space, and by asynchronously measuring and exchanging gradients between address spaces.

## Bugs / Todos

* If the bandwidth between the GPU and host is set too high, machines seem to hang.
* Incorrect hyper-params schedule in the distributed case. The total count of iterations needs to be tracked, maybe through the monitoring tool.
* Thread-local Caffe singletons are not destroyed, we need to design a proper shutdown strategy.