Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work presents Petals, a system designed for inferencing and fine-tuning large-language models over distributed, commodity hardware.     This enables users to leverage large, pre-trained models without having exclusive access to high-end hardware, and instead being able to leverage the collective inferencing capability of many systems across the internet.   The system uses pipeline-based model parallelism, distributing model layers across nodes, and both server and client caches to restore state during server failures.  They build the system and evaluate its performance through a failure microbenchmark and illustrate performance on larger clusters with different emulated network conditions.  In that scenario, Petals is about 10x faster compared to offloading on a single GPU.
strength_and_weaknesses: Strengths
* Great problem, high impact, fun read 
* Technically interesting system architecture with a functional implementation  
* Evaluation in the wide area, compelling numbers wrt an offloading baseline

Weaknesses
* The description of the fault-tolerant load-balancing algorithm is insufficient.  The stability of the N-player greedy approach needs to be addressed.  The performance model is not described nor its accuracy evaluated.   
* It isn't clear how Petals initially groups layers to the servers or if layer groupings can change.   
* The evaluation does not empirically explore actual server failures or network partitions (to the client, between servers).   I.e., there are no results bolstering the core contribution (I comment below on why Table 1 is a great microbenchmark but insufficient in this regard). 
* Explores uniform network bandwidth and latency, but it's unclear what this is really testing in the Petals architecture.   
* Compares to offloading but should compare to DeepSpeed in the wide area under conditions that illustrate Petals benefits. 
clarity,_quality,_novelty_and_reproducibility: I really enjoyed this paper.  It's a great problem and creating a system for inference/fine-tuning on LLMs would be useful.   Overall the description of the related work and system architecture were clear.   

I really appreciated the detailed algorithms (1,2,3) in Section 3.2.   However, there are a few areas where the details were perhaps insufficient.   Section 3 could use a few system diagrams to illustrate how layers are assigned to stages and stages to servers, as well as the client/server caches, what they hold, and when/where their values are sent. 
* Section 3.3 load balancing:  How does the system measure throughput?  Continuous updates to DHT?  Is there a notion of total server load (i.e., the amount of memory -- can a new stage even fit on this server?)? 
* It isn't clear how consecutive layers are initially assigned to stages.   Is each layer considered in complete isolation?  Doesn't seem so from experiments.   Is it static?  Should you pack the NA onto the fewest, closest servers?  
* Do you consider re-locating the client to be centrally located wrt to its servers?   It's essentially a controller node -- it doesn't have to be co-located with the user initiating the task. 
* The load-balancing scheme not only allows recovering servers to assign themselves layers, but operational servers to reconsider the layers they serve.   It's possible that the system could easily oscillate as some nodes decide to host new layers, creating a cascade of changes.  Determine if the system is stable and emperically demonstrate. 
* You state "they switch layers until the throughput becomes near optimal."  That claim needs to be validated.  
* it's great to support pipelining of activations to the next downstream stage and the client.   However it makes failure handling more complicated.  What happens if the client becomes partitioned from the next server but the upstream server does not?  How much does this help performance? 

Evaluation 
* Really like the microbenchmark in Table 1.  However you state that you simulate failures by resetting pipeline stages.  That might internally be different than Algorithm 1 taking the RPC_Failed path.  In other words, a live server with a "failed stage" has a lot of information and retains connectivity, where a disconnected failed server does not.   So, is re-balancing occuring here or not?   Or are we purely looking at the costs of restart vs. recompute?   I think it's the later, but be clear. 
* You run across the wide area and that's great.  But were there any failures?  Were there any re-balancing actions taken? Kill some servers, create some network partitions, create some bad connections -- let's see how Petals really works. 
* It's great to compare to a single GPU with offload.  But it seems that running DeepSpeed in the wide area (or even on the emulated network (maybe something like https://www.edge-net.org)) is another important comparison point.  
* The model to determine load balancing wasn't evaluated - does it work? 

Other items:
* This kind of work is related to different strategies for fault tolerant stream processing.  It might be interesting to see if there are related techniques (checkpointing) and to note what's new here.   Maybe https://raulcastrofernandez.com/papers/sigmod13-seep.pdf is a start. 
* Please define block, layer, and stage.  Are they the same, are they different?  Try not to give extra mental work to the readers ;) 
* S4.1 listing the three strategies it would be helpful to be clear what "restart" means in each line.   Restart the entire job from the start?  Also, be clear about step.  Is that one step through all stages or a forward step on one stage?   
* S3.2 P8 is hard to understand.  Maybe just a bit more?  
* S3.2 P7 fix "sends the to the subsequent" 
* S3.3 P1 fix "server joins OR leaves" ? 
* The language in S3.2 P1 repeats the "every device can act as client and server"


summary_of_the_review: Interesting, motivated, impactful problem whose solution was evaluated on both local and wide area networks for wide scale inference of LLMs.  The work could use further details and experiments to validate the claim that the system works well under network and node failures.  

[Post Author Response] I've reviewed the author's responses and paper changes.  They have addressed a number of my concerns, there remain open questions around system operation at scale (vis-a-vis load balancing).   Some could be addressed by describing the limitations of the current load balancing heuristics and analysis.  Even with these drawbacks, the work could be impactful.  I've adjusted the rating from marginally below to marginally above.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents Petals, a system for inference and fine-tuning of large language models on distributed machines over the internet. The system mainly includes the following techniques:

- Pipeline parallelism across geo-distributed devices.
- Fault tolerance via storing an extra copy of pipeline communication data within the client.
- A dynamic load-balancing algorithm that eliminates performance bottlenecks.
- Using adapters for fine-tuning and storing the adapter parameters in the client for fault-tolerant fine-tuning.
- Quantization and compression to reduce memory footage.

The authors evaluate the system on a smaller BLOOM model for fault tolerance performance and test on the large BLOOM-175B model with multiple hardware settings to test the latency of the proposed system.
strength_and_weaknesses: ## Strength

1. The paper proposes a valid combination of multiple techniques for efficient inference and fine-tuning of LLMs, including pipeline parallelism, adapters, quantization and compression.
2. The engineering behind the system is non-trivial and I appreciate the authors for open-sourcing their work. I believe the work will be beneficial for researchers with access to geo-distributed machines.

## Weakness

1. Pipeline parallelism is a well-studied technique for large language models. The way this paper applies pipeline parallelism is not new compare to previous works.
2. The dynamic load balancing (i.e., schedule which layers to run on each GPU) are not evaluated in the experiments. How does this algorithm adapts to different hardware configurations?
3. Comparing the proposed setting with the offloading techniques is not a completely fair comparison: The offloading setting only has a single GPU, but the geo-distributed setting has many more GPUs and thus much higher total computational power. 
4. A more important comparison is to compare the efficiency of this method with the setting where the GPUs are within a single datacenter and the optimal performance in that case.
clarity,_quality,_novelty_and_reproducibility: The paper is well-written and easy to understand in general. With the provided code, I believe the results of the paper can be reproduced by researchers or engineers with access to geo-distributed devices. However, I have concerns on the novelty of the paper in the weakness discussion above. Below are some of my extra questions to the work:

1. Page 3: “Counter-intuitively, we found that inference is more challenging than fine-tuning for cost-efficient setups.” Is this due to inference is an autoregressive iterative decoding process? Please elaborate.
2. Page 7: “Servers can run backpropagation through their layers and return gradients with respect to activations”: During fine-tuning, how does the server deal with the intermediate activation from the forward pass? Do you always turn on gradient checkpointing, or do you cache the intermediate activation during the forward pass?
3. Table 1: How is failure rate defined exactly?
4. Table 1: Why does Algorithm 1 perform worse than recompute with 128 tokens and 1e-2 failure rate?
5. Do you only focusing on the case where there is only one client? How do you handle multiple clients?

Nit:

Page 1: “they are still difficult use due to” → “they are still difficult *to* use due to”
summary_of_the_review: The paper proposed a solid system for geo-distributed inference. The work includes solid system engineering but lacks enough novelty for a research paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes PETALS, a system that enables inference or finetune processing of large language models (LLM) without the cost of HPC hardware, but rather by sharing a large pool of consumer-grade hardware. PETALS uses pipeline parallelism to partition LLM layers over the geographically distributed devices and proposes a fault tolerant algorithm for inference and fine tuning to address reliability issues. The paper presents some evaluation of the fault tolerance algorithm and performance comparison to offloading. 
strength_and_weaknesses: Although, pre-trained LLMs are now freely available for download the hardware cost of inference and finetuning computation is still quite prohibitive and is a barrier to broad accessibility of AI advancements. Thus, the paper is tackling an important problem. The proposal of a community-approach that involves pooling consumer-grade resources together to inference and finetune LLMs is also reasonable and applicable to a broad class of users.  The presented algorithm and implementation details are useful in understanding some of the technical challenges. 

My main concern is the evaluation, which I feel does not present a complete and fair assessment of PETALS. Below are specific examples. 

1. Overhead of fault tolerance (4.1)
    a) What are the runtime and memory costs of client and serving caching, and how they scale with increasing client and server counts?
    b) When clients don't have GPUs, does this mean client-server communication involve GPU/CPU copying on the server?
    c) Are the runtime and memory costs of fault tolerance reflected in the results in 4.2 and 4.3?
    d) Table 1 only shows fault tolerance evaluation for inference. Is the fault tolerance provided for fine tuning?
    e) In Table 1, PETALS is the best option in only 2 of the 8 scenarios. Caching with restarts wins 5 times, No caching wins once. This is not very convincing for PETALs. 

2. Comparison to Offloading (4.2)
   a) Using 1 A100 for Offloading and 3 A100 for PETALs does not seem like a fair comparison. Can you elaborate?
   b) The theoretical latency numbers of Offloading suggest that 16-bit weights are being used rather than 8-bit weights, is this correct?
   c) Are the A100 GPUs connected by PCIe or NVLINK?
  d) Depending on the response to the preceding 3 issues, a fair evaluation could mean scaling the Offloading performance by up to 18X: 
    3X for GPU count, 3X for parallelizing weight fetching from CPU, and 2X for 8-bit weight quantization. 
  e) Why are there no forward pass results for the multiple client section of Table 2?
 
clarity,_quality,_novelty_and_reproducibility: The paper is generally well written and easy to read. The paper has several novel aspects and interesting technical details. The evaluation could be improved as described earlier. 
summary_of_the_review: While I think PETALS would be a useful system to many users, the current evaluation makes it difficult to understand how well it would function and perform in real life scenarios. Moreover, the previously noted evaluation issues makes it difficult to know the scenarios that PETALS would be preferable to caching with restarts and offloading.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors proposed a method to perform inference na fine-tuning of LLMs over in a distributed manner over the internet, based on the fact that activation communication between geo-distributed devices could be faster than offloading to local memory for really large LMs. The authors developed a fault-talent algorithm to handle the error and proposed some system design guidelines. Experimental results show that the proposed method can outperform caching/recomputing under higher failure rates and has a higher throughput than RAM offloading. 
strength_and_weaknesses: Strength:
1. Firstly, the paper is generally well-written and easy to follow. 
2. The observation that transferring activation is faster than offloading weights in LLMs is interesting. 
3. The proposed method addressed an important question and allowed communities with lower-end GPUs to be able to serve and tune LLMs. There could potentially be a new business model based on collective inference.

Weakness:
1. I am not very familiar with distributed system research. But from my understanding, fault tolerance in a distributed system has already been well studied. It seems the proposed method only applies the techniques to LLM inference and does not have a significant novelty. 
2. From my understanding, the method can only work when the model is large enough (as the authors mentioned, >100B). Could you please show how it works for smaller LMs? What model size is the turning point?
3. When comparing the baseline of offloading, did you apply the same quantization technique to reduce bandwidth requirement? 
4. Did you implement pipeline parallelism to further improve the throughput?

clarity,_quality,_novelty_and_reproducibility: The clarity, quality, and reproducibility are good.
Regarding novelty, it might borrow part of the contents from the existing research on fault tolerance in distributed systems. 
summary_of_the_review: Please see the Strength and Weakness section. I am not 100% familiar with the literature on distributed systems. And I am open to raising scores given a convincing reply.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper develops a system for inference and fine-tuning large language models on distributed machines. The system, Petals, includes a few components such as pipeline parallelism, fault tolerance, load-balancing, adapter-based finetuning, quantization and compression, etc. 
The system shows improved speed on large clusters, e.g., 10x faster compared to offloading on a single GPU. The reviewers have concerns about the evaluation, such as the evaluation of dynamic load balancing, unfair comparison with single-GPU offloading, comparison with GPUs in a single datacenter, overhead of fault tolerance, and others. The author response addressed some of the concerns. 
justification_for_why_not_higher_score: Reviewers expressed a few concerns about insufficient evaluation.
justification_for_why_not_lower_score: The system is interesting and the problem is timely.
Decision: Reject