Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper provides a model to optimize large DL models. It takes into account TMP, DMP, and PMP. Moreover, the proposed model accounts for heterogeneity that exists across devices. 
strengths_and_weaknesses: Strengths
—-

* models device heterogeneity 
* performs better than state of the art


Weakness
—-

* model is not generic. Lot of heuristics that have been integrated; thereby, exploring few possible optimizations
* For example, why was latency differences among devices not modelled. It is well known that both bandwidth and latency play a role
* Similarly, why not consider different generations of GPUs. Is not that more realistic?
* Does not really account for virtualisation and multi-tenent environment. 
* moreover, the pruning strategies are not really clear. Overall it seems to me that most of the optimization can be specified by a rule-based system
questions: Please address the weaknesses mentioned above.
limitations: * Major heterogeneity that exists today in clusters are in terms of GPU types, e.g., use of A100, V100 etc. Paper falls short of its claims

* It is unclear as to why only select few device characteristics such as bandwidth was used. 

* Overall it is bunch of heuristics rather than a global and generic optimization framework. Perhaps the paper is more suited for a system’s conference such as OSDI/Eurosys/ATC
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a method to automatically determine the parallel strategy that includes data parallelism (DP), pipeline parallelism (PP), and tensor model parallelism (TMP) over a distributed runtime with heterogeneous computation and communication resources.
strengths_and_weaknesses: Strengths:

- Automatically determining the combined parallel strategy is an interesting and important problem for distributed machine learning considering the advanced of giant foundation models. 

- The formalization of the problem and the proposed cost model is reasonable.

Weaknesses:
- The presentation of the methodology part can be further polished; some of the details are difficult to follow for the first time. For example, a table defining all the variables demanded in the cost model will make Section 3 easier to follow; and there should be a clear discussion about how communication and computation are considered as overlapped execution; perhaps a Gantt chart will be helpful to illustrate this. 

- The optimization procedure is not clearly stated in Section 3.6. In fact, Section 3.6 is just a sketch instead of a concrete discussion of the technique. Many details are missing for optimization. 

- The experiments are conducted on a relatively small GPT-2 model with a small batch-size; the cluster scale is also small with 4X4 GPUs in total. In fact, the practical challenge for automatic layout is for very large clusters with hundreds or even thousands of GPUs. In the current scope of experimental results, it is difficult to verify the effectiveness of the proposed method.   
questions: - In line 116, does the tensor refer to output activations from layer i? 

- Would it be possible to be more concrete about the optimization procedure? For example, can the dynamic programming approach explore all the space? What is the complexity of the optimization? 

- Would it be possible to provide the execution time of the optimization procedure in the experiments? 
limitations: It is helpful to note the limitation on the lack of memory constraints in the cost model.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents a method to automatically find the optimal parallelization strategy for large model training. Unlike previous frameworks, the proposed model considered the heterogeneity of the model and hardware. The parallelization problem is defined as the parallelism degrees, micro batch size, device assignment, and the arrangement of the layers in the pipeline. The authors built a cost model to estimate the cost of the candidate strategy. The parallelism degrees and micro batch size are selected through enumeration. The device assignment is solved by applying a heuristic while the pipeline arrangement is found through dynamic programming.
strengths_and_weaknesses: ## Strength

1. This paper present a detailed cost model for parallelization strategy on heterogeneous models and hardware. The difference in hardware is considered as different bandwidths and the profiling result of the layer execution time.

2. The pipeline arrangement is modeled as a dynamic programming problem and given a determined solution. 

3. The ablation study shows the effectiveness of each component optimized by the proposed method.


## Weakness

1. **The accuracy of the cost model.** The evaluation focuses on the iteration latency. But as a major contribution of this paper, it is also important to understand the accuracy of the cost model and how will it affect the strategies found by the method. Based on what is shown in Figure 2, it seems that the estimation for most strategies is optimistic, especially for the optimal one indicated by the estimation. 

2. **Discrete optimization.** The authors decompose the strategies into four different subproblems and separately optimize for each problem. Will this lead to a suboptimal solution compared with a joint optimization method?
questions: 1. Could you provide extra information on the accuracy of the proposed cost model? For example, the correlation coefficient against the real latency. Could you also indicate how will the accuracy of the cost model affect the result?

2. Could you comment on whether it is possible to perform joint optimization and how does it compare with the discrete optimization of each subproblem?
limitations: The author discussed the limitation in terms of the lack of memory model.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper presents a strategy in model-parallel setting, where the model is being trained in a cluster. The strategy considers heterogeneity of computational resources, diverse characteristics of different layers, communication costs in networks, and uses a dynamic programming algorithm to intelligently assign layer-stages.

strengths_and_weaknesses: Strength:
1) A dynamic programming-based algorithm is prposed to find out the optimial placement strategy that is significantly time-efficient that bruteforce exhausive search;
2) The paper proposes an analytical cost model that saves the cost of estimating the run time of each layer on actual hardware;
3) Experiments show that the strategy is highly effective and provides concrete speedup for end-to-end workloads.

Weakness:
1) The paper doesn't present comprehensive ablation study of the effectiveness of the cost model.
2) The paper seems to overlook some latest advances in distributed training and model parallelism, for example, Alpa [1] that considers homogeneous setting but still formulates the problem similarly, while having heterogeneity seems to be a plain extension. Would love to hear from the reviewer about this particular comparison.

[1] Zheng L, Li Z, Zhang H, Zhuang Y, Chen Z, Huang Y, Wang Y, Xu Y, Zhuo D, Gonzalez JE, Stoica I. Alpa: Automating Inter-and Intra-Operator Parallelism for Distributed Deep Learning. arXiv preprint arXiv:2201.12023. 2022 Jan 28.

questions: 1) The reviewer is interested in more study on how accurate the cost model is, for example, what if we replace the analytical approximation with an oracle cost model?
2) Compared with previous work like Alpa which uses homogeneous setting, how far does the proposed method go besides adding more terms in the dynamic programming equaltion?
limitations: As the author stated, memory footprint is right now not under consideration. However, the reviewer does not consider this as a significant drawback.
review_title: null

Meta-review: metareview: This paper presents AMP, a method to automatically find the optimal parallelization strategy for large model training on heterogeneous compute resources taking into account the model architectures and cluster setups. The method is based on a model to estimate the cost of candidate strategies. A combination of heuristics and optimization techniques is used to determine the degree of parallelism, the micro batch size, the device assignment and the pipeline arrangement. 

The reviewers agree that the paper addresses a relevant and timely challenge for model training, it proposes a solution that is more general than previous approaches, and it shows promising empirical performance. Most of the questions about the method and concerns about the presentation could be addressed in the rebuttal. The overall assessment of all reviewers is positive, I thus recommend acceptance, counting on the authors to take the feedback into account and incorporate the author response into the revision.

However, I want to emphasize that I think two concerns that came up are important and investing more in them could strengthen the paper a lot. 
- First, the cost model is at the core of the contribution and underlies your parallelization strategy. So a more comprehensive ablation study (as requested by two reviewers) would be appropriate. The  spearman correlation numbers that you added in response provide valuable information, but they still leave many questions unanswered of how the cost model performs beyond the experimental setup you are currently using and how it behaves along different parameters in the cluster configuration. Such an ablation study could be evaluated independently of the benchmark results. Any effort along this direction to provide additional insights would be helpful and appreciated.
- Second, as you acknowledge experiments are at relatively small scale. However scalability is an important aspect of modern systems that is not discussed sufficiently in the paper. I understand that hardware resources might not be available and there is a justified use case also at smaller scale, but you should be forthcoming about the potential limitations of your model and the search algorithm that come with the size of the cluster you apply your algorithm to. Being clear about the scope of the work helps the reader  judge the applicability of your solution in practical problems.
confidence: Less certain
Decision: Accept