Before revision:  We discuss various implementations of pipeline parallelism and propose a novel  schedule that can improve throughput by more than 10\% with comparable memory footprint compared to previously-proposed approaches.
After revision:  We discuss various implementations of pipeline parallelism and propose a novel interleaved pipeline parallelism schedule that can improve throughput by more than 10\% with comparable memory footprint compared to previously-proposed approaches.