Kati8/ICLR_2023_2025 huggingface dataset:
authors: Yeming Wen
authors: Swarat Chaudhuri
reviews: 
Comments: LoRA, a widely used technique for fine-tuning a small number of parameters in foundation models, exhibits a weakness in batched inference settings where each request in the batch requires a unique adapter.
In such a scenario, batched inference using LoRA becomes sequential and inefficient. This paper proposes a variant of LoRA, called fast LoRA (FLoRA), which utilizes a parameterization that enables minibatch computations to be performed using matrix multiplications. This makes it efficient to perform batched inferences with distinct adapters per request. 
The paper presents a computational analysis demonstrating that FLoRA can achieve improvements in both throughput and latency compared to LoRA for scenarios involving low-rank and small model dimensions. 
The paper presents empirical results demonstrating the advantages of FLoRA over LoRA when using StarCoder (Li et al., 2023) as the foundation model. On multilingual code generation and speech recognition tasks, FLoRA achieves similar performance to LoRA and outperforms IA3.
Strengths: * Proposes an alternative to the LoRA approach that is efficient for batched inference with distinct adapters per request.
* Presents an analysis demonstrating the conditions under which the proposed approach can outperform LoRA.
* Demonstrates using the StarCoder 15B LLM that FLoRA can double the throughput (halve the latency) in a low-rank setting when diverse adapters are required for incoming examples.
* Shows that FLoRA yields similar results as LoRA on multilingual code generation and speech recognition tasks.
Weaknesses: * Some parts of the paper are not clear (see comments below).
Recommendation: 8: accept, good paper
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 4 excellent
Clarity: 3 good
reviews: 
Comments: The paper proposes FLoRA, which allows each example in a minibatch to own unique low-rank adapters. FLoRA encourages efficient batching of serving various requests, retaining performances of LoRA with throughput improvement and latency reduction in low-rank settings.
Strengths: 1. The orientation is clear. It can important to equip language models with various task-specific adapters for diverse requests. The overall idea is well-motivated.
2. The formulation is clear and analysis of computational consumption is in detailed.
Weaknesses: 1. If each example in a minibatch has its own adapters, the overall performance is expected to overcome LoRA, however, it's almost the same as LoRA. So the "performance bottleneck in scenarios requiring personalized, task-specific adaptations for each incoming request" isn't largely solved.
2. The whole mechanism and the algorithm isn't mentioned clearly. e.g., how to choose the batch size for real situations, how to make each example corresponding to its appropriate adapters during inference. The paper over-concentrates on Fomulation and Computational Efficiency, while the high-level algorithm--the whole process is not quite clear.
Recommendation: 8: accept, good paper
Reviewer confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 3 good
Clarity: 3 good
reviews: 
Comments: The paper builds up on the Low-Rank Adaptation (LoRA) framework to fine-tune foundation models, by introducing fLoRA, which allows distinct adapters for different task-specific requests within the same batch. The authors empirically demonstrate that their approach preserves the advantages of LoRA in terms of accuracy on multilingual code generation and speech recognition tasks, while facilitating a higher throughput and lower latency.
Strengths: The paper clearly introduces the problem and the contributions compared to the state of the art. The contribution is significant to cope with practical challenges of using foundation models in real-time serving scenarios, especially when considering world-wide incoming requests.
The paper looks theoretically and technically sound and the presentation is clear, well framed in the context, and easy to follow.
Weaknesses: I don’t find major weaknesses. Minor comments are indicated in the following section.
Recommendation: 8: accept, good paper
Reviewer confidence: 2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 3 good
Clarity: 3 good
reviews: 
Comments: The paper propose a new low rank adaptation technique based on a generalization of IA3.
Essentially the adaption changes from LORA: W = W0 + BA to FLORA: W = W0.*BA
This allows to pack in a batch many different adaptors per input or even per chunk efficiently.
Strengths: The paper presents several strong points.
The proposed approach improves latency and throughput as well as a theoretical cost estimation.
Several model sizes from starCorder and LLama 2 are considered for throughput and latency estimation.
The accuracy of the proposed method is similar or better to that of LORA and IA3 and report improvements/checks on several models such as Llama2, whisper or starCoder.
Weaknesses: The approach requires re-adapting the models that have already been adapted with LORA to leverage the improvements.
There is a breaking point where FLORA doesn't improve over LORA effectively. Intuitively, there is at least 4 factors for this: the model, the gpu architecture, the rank of the adaptation and the batch size . The rank is taken into account but it is not very clear how the other elements will come into play in practice. Eq 7 claims only important factors are the dimension of the multiplication the constants for MM and BMM and the rank. However, it is difficult to understand why this should be the case for batch size 1 in contrast to a larger batch size.
Computing some plots in this area would have been very helpful to grasp how the theoretical analysis transfer to the practical scenarios..
Another example of this would be computing per token and example adapters , which is the extreme case. It would have been interesting for latency and throughput curves to see such an extreme case, even though there is no such a real task. 
The section 3.1 is confusing in its current form and a rewrite paying attention to the Matrix and elementwise operations would improve readability. 
Given the constrains of the approach regarding the low-rank dimension, the applicability of the approach is limited to some specific scenarios which could have already been taken care on the base LLM pretraining. For instance, for the multilingual case the models could have already specific sparsely activated components given the language category or the programing language from the beginning.
Recommendation: 8: accept, good paper
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 4 excellent
Clarity: 3 good