diffdist (pypi): Provides differentiable communication primitives such as Send/Recv/Gather etc for pytorch. Autograd can backpropagate through those, making distributed model parallel models easy to implement.
Keywords: torch|distributed|autograd|model|parallel|differentiable|backpropagate