Before revision:  In this work, we show how to compose different types of parallelism methods (tensor, pipeline, and data paralleism ) to scale to thousands of GPUs, achieving a two-order-of-magnitude increase in the sizes of models we can efficiently train compared to existing systems.
After revision:  In this work, we show how to compose different types of parallelism methods (tensor, pipeline, and data parallelism ) to scale to thousands of GPUs, achieving a two-order-of-magnitude increase in the sizes of models we can efficiently train compared to existing systems.