Before revision:  New methods of model parallelism such as tensor and pipeline parallelism have been proposed to address these challenges ; unfortunately , naive usage leads to fundamental scaling issues at thousands of GPUs due to various reasons, e.g., expensive cross-node communication or idle periods waiting on other devices.
After revision:  New methods of model parallelism such as tensor and pipeline parallelism have been proposed to address these challenges . Unfortunately , naive usage leads to fundamental scaling issues at thousands of GPUs due to various reasons, e.g., expensive cross-node communication or idle periods waiting on other devices.