PALMBENCH: A COMPREHENSIVE BENCHMARK OF COMPRESSED LARGE LANGUAGE MODELS ON MOBILE PLATFORMS

## \thesubsection Multi-Turn Questions

MT-bench [MTBenchArena_2023] employs a predefined multi-turn question set to assess conversational models across turns for instruction following and dialog coherence, critical capabilities for human-like interaction. Each sequence begins with an initial question, directly followed by a contextual follow-up question. Model responses at each turn are evaluated against human-generated ground truth answers using GPT-4 scoring.
Our goal was to tailor benchmarking for mobile insights rather than fully evaluating model capabilities to inspire observations for LLM deployment on mobile devices. We just test two tasks tailored for mobile platforms.
Task 1 evaluates the extraction ability of an LLM that summarizes a paragraph of text, and Task 2 prompts the model to write a paragraph of text.
To evaluate model performance on more complex tasks across platforms, a predefined test set was constructed using two challenging extraction and writing tasks from the MTBench multi-turn benchmark dataset. (^1^11\urlhttps://huggingface.co/spaces/lmsys/mt-bench).
Figure \thefigure shows the results of memory usage and benchmark scores of two tasks.
Profiling showed the GPU operates near saturation with over 95% utilization on MTBench, while memory increases slightly for multi-turn dialog.
However, the MLC Chat framework does not currently utilize neural processing unit (NPU) inside mobile SoC, missing optimization evidenced by 0% utilization of NPU during inference.

Figure: Figure \thefigure: Memory usage and scores across models ((Llama-2-7B, RedPajama-v1-3B and Vicuna-v1-7B)) and platforms (IP12-iPhone 12, Pixel 4, S22U-S22 Ultra and Pi-Orange Pi) in MT-Bench evaluation.

Task 1 and Task 2 are more complex than the Simple Q&A task and are anticipated to use more resources and consume more power.
Figure \thefigure and Figure \thefigure show the experimental results of power consumption (mA), throughput (tok/s) and latency (s) while running extraction and writing tasks from MT-Bench test questions.
Results revealed increased energy consumption and latency across all platforms when evaluating models on dialogs with multi-turn questions.
Throughput remained relatively constant for Vicuna-v1-7B, while Llama-2-7B and RedPajama-v1-3B exhibited throughput gains on longer conversations.

Figure: Figure \thefigure: Task 1 - Extraction evaluation across models ((Llama-2-7B, RedPajama-v1-3B and Vicuna-v1-7B)) and platforms (IP12-iPhone 12, Pixel 4, S22U-S22 Ultra and Pi-Orange Pi).

Figure: Figure \thefigure: Task 2 - Writing evaluation across models ((Llama-2-7B, RedPajama-v1-3B and Vicuna-v1-7B)) and platforms (IP12-iPhone 12, Pixel 4, S22U-S22 Ultra and Pi-Orange Pi).

Figure \thefigure and  \thefigure indicate that all three models require increased resources, particularly memory, in multi-turn conversation tasks. While resource utilization remains consistent on the iPhone, significant variations are observed across Android platforms. Orange Pi [OrangePi] runs the model via a command line interface, with less memory utilization.
The Task2-Writing exhibited longer latency than Task1-extraction, as text generation produces more words and tokens. Despite increased latency for multi-turn versus simple conversation of Simple Q&A, RedPajama-v1-3B maintained relatively fast response times for answer generation across tasks.

## \thesubsection Quantization

We evaluate how various LLM quantization schemes impact resource consumption and user experience on mobile devices, of which observations serve as foundational insights for crafting optimized strategies for LLM deployment on these devices.
Quantization can reduce memory and accelerate inference.
MLC [MLC_LLM] provides options for quantization with quantization 0-bit (q0f16), 3-bit (q3f16), 4-bit (q4f16), and 8bit (q8f16).
Llama-2-7B and Vicuna-v1-7B’s resource demands surpassed our testbeds’ capacities, particularly in memory. For example, the 8-bit and 0-bit versions of Llama-2-7B need 8.93GB and 15.13GB RAM, whereas iPhone 12 Pro and Pixel 4 offer just 6GB RAM.
Quantization evaluations were performed using the resource-efficient RedPajama-v1-3B model on a Samsung S22 Ultra testbed.
Figure \thefigure illustrates the performance metrics (latency, memory utilization, and power consumption) of RedPajama-v1-3B under various quantization schemes on the S22 Ultra. Notably, the 3-bit (q3f16) scheme demonstrates decreased memory requirements and quicker inference, benefiting from reduced memory bandwidth and storage needs.
In contrast, 0-bit (q0f16) exhibited substantially higher latency and near complete memory usage, representing an infeasible quantization scheme given current mobile device constraints.

Figure: Figure \thefigure: Results of evaluation across different quantizations, 0-bit (q0f16), 3-bit (q3f16), 4-bit (q4f16) and 8-bit (q8f16) in latency (s), memory usage and power consumption.

The precision and answer quality will determine the user experience and quality of the LLM chatbot applications.
We evaluate these four quantization schemes using MT-Bench,
as depicted in Table \thefigure.

**Table \thetable: The scores of MT-Bench in extraction and writing tasks across various quantization schemes**
|  | 0-bit (q0f16) | 3-bit (q3f16) | 4-bit (q4f16) | 8-bit (q8f16) |
| --- | --- | --- | --- | --- |
| Task1-Extraction | 6.4 | 1.4 | 6 | 6 |
| Task2-Writing | 7.6 | 3.4 | 6.8 | 7 |

Results in Table \thefigure show 3-bit quantization provides minor resource savings but falls short of delivering a satisfactory accuracy, while 8-bit incurs overhead without much accuracy gains versus 4-bit.
The 4-bit scheme q4f16 strikes the optimal balance for resource utilization, latency, efficiency, and precision on mobile platforms, achieving a 34.1% improvement in overall scores compared to 3-bit quantization.

## \thesubsection Model Size

Besides quantization, model size significantly impacts performance and resource utilization. Smaller models with fewer parameters reduce memory requirements and enable faster inference than larger counterparts, which means they are more suitable for devices with limited computational resources.
Larger models demand increased resources but can improve performance and provide more precise answers, while smaller models are more efficient yet may sacrifice accuracy.
An appropriate tradeoff should be made considering the target platform’s capabilities and user requirements.
Table \thefigure shows RedPajama-1v-7B with extensive parameters has better MTBench scores but higher memory and latency than its smaller counterpart, resulting in reduced throughput.
RedPajama-1v-3B yields a slightly reduced MT-Bench score of 6.4 but conserves up to 19.1% in memory usage.

**Table \thetable: The comparison between RedPajama-1v-3B and RedPajama-1v-7B in terms of overall memory usage, latency, and throughput.**
|  | RedPajama-1v-3B | RedPajama-1v-7B |
| --- | --- | --- |
| Memory Usage | 3.86GB | 5.39GB |
| Latency | 12.3s | 35.5s |
| Throughput | Prefill: 24.8 tok/s Decode: 11.9 tok/s | Prefill: 7.84 tok/s Decode: 3.76 tok/s |
| MTBench Score | 6.4 | 7.2 |

## \thesubsection Deployment Strategy

Based on our experimental observations, we propose deployment strategies for on-device LLMs, outlining a decision model along three dimensions to guide mobile deployment.
In utilizing the MLC framework to deploy a pre-trained model, one can adjust the model architecture, quantization scheme, and model size. These adjustments strike a balance between performance and efficiency, representing the three key dimensions of our decision model.
Each dimension corresponds to distinct user needs. For instance, some users are not sensitive to latency and don’t mind waiting longer for a more accurate response. Others are conscious of their phone’s battery consumption and would prefer the LLM not to drain too much power. Furthermore, some users manage work-related tasks on their mobile devices and would not appreciate the LLM chatbot occupying the device’s resources.
As illustrated in Figure \thefigure, Llama-2-7B outperforms in delivering more accurate responses, albeit at the expense of longer processing times and increased resource consumption over RedPajama-1v-3B and RedPajama-1v-7B. This makes Llama-2-7B suitable for users who care more about answer precision over response time and have a greater tolerance for latency.
For individuals using devices with limited resources, especially those with less than 4GB of memory, RedPajama-1v-3B with 4-bit quantization is a preferable option. The 3-bit quantization scheme q3f16, though offering slight savings in resource usage, latency, and power consumption, falls short of delivering a satisfactory user experience.
4-bit quantization q4f16 should be the best option to strike the tradeoff among resource, latency, energy efficiency, and precision on mobile devices.
On the other hand, Vicuna-1v-7B underperforms regarding resource efficiency and MT-Bench scores across all three evaluated models, leading us to exclude Vicuna from our deployment strategy.
As shown in Sections \thefigure and Section \thefigure, optimized deployment strategies can improve memory efficiency by up to 19.1% and boost scores by 34% on MTBench for on-device LLMs. Appropriate configuration and sizing can yield substantial resource savings and accuracy gains.
Additionally, the evaluation highlights substantial platform differences as depicted in Figure \thefigure. iOS exhibits lower latency and higher energy efficiency while maintaining greater throughput compared to other mobile operating systems.
In our experiments, we noted a consistent reliance on GPUs across all platforms for model inference. To enhance efficiency, developers can leverage the full capacity of hardware resources. For instance, MLC has incorporated support for Qualcomm Snapdragon’s Hexagon DSP for inference, offering an alternative to shift some tasks from an overburdened GPU to the DSP.
Additionally, our observations indicated that the NPU on Apple’s A14 consistently showed 0% utilization, suggesting no tasks were executed on it. Given that the iPhone’s GPU usage nears 100% during LLM chatbot operations, iOS developers might consider offloading model inference tasks to the NPU.

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: ### Motivation

* The authors suggest in the introduction that "running LLMs locally" can lead to "increased reliability". Personally and from experience, I would not be that confident on making such claims, i.e. that cloud deployments are less reliable that deploying locally, especially given the infancy of the evaluated frameworks. I would appreciate some more substantiation here from the authors.

### Methodology

* Wrt methodology and experimental setup, the paper misses substantial information that hurt the overall reproducibility potential. Such omissions include:
    - Operating system and framework versions
    - Automation of iOS interface
    - How different components in Figure 1 actually coordinate and communicate.
* Would the authors be willing to open-source their benchmarking system?
* It is unclear whether the authors have converted the models themselves or have used the pre-compiled versions offered in GGUF and MLC repositories.
* How do the authors ensure that the profiling methodology does not interfere with the behavior of the experiment? Also, how do the authors isolate the power lanes from the USB communication?
* The authors claim that they are using a "professional USB power meter for accurate power consumption measurements". However, it is not clear how this yields the information needed, as devices are battery powered and not USB-powered. As such, the power draw from the USB does not yield any information related to energy consumption from a workload on device.

### Evaluation

* The evaluation does not integrate variance metrics. Should it be assumed that the experiments have run once?
* What is the decoding strategy and hyperparameters that the authors use during evaluation?
* With respect to temperature, does the average represent "average over time" or "average over surface"?
* A missing dimension that is worth exploring in such a paper is the tradeoff between size due to parameters and due to quantization (or another compression method). For example, is it better using a lower-bitwidth llama-3.1-8b model on 4bits or a llama-3.2-3b model on 8 bits?
* What is the bitwidth used in Figure 2 across models?
* In Figure 3, the GPU utilization across phones is quite large, which comes in contrast with the "memory bounded" workload claim of the authors. I would appreciate some more insights here.
* §4.2: "3-bit quantization results in lower CPU and GPU usage [...] decreased data transfers [...] reduced inference workload": I am not sure this claim is correct or substantiated properly.
* A great source of information to the reader, also for showcasing the memory-boundedness of the workload would be to plot a roofline model for devices/models.
* The authors claim that iPhones provide significantly higher throughputs compared to other devices. This may not be painting the whole picture, as it is not clear whether this throughput can be sustained compared to the actively cooled Jetson Nano for instance.

### Related work

* The authors claim that prior work (MELT) has examined resource utilization and energy efficiency in a limited manner, and did not explore GPU workloads. However, this is not true. 
    * Additionally, the authors do not run llama.cpp on iOS, which prior work has done.
    * Palmbench does not measure power consumption via hardware probes on phones and do not measure it at all on edge devices.
    * Palmbench does not report on prefill rates, which prior work does.
    * Palmbench does not integrate high-end edge devices (e.g. Jetson AGX) and different power profiles, which prior work does.
* Moreover, the authors have unfortunately missed other related work in the domain of on-device LLM benchmarking [a-c].

[a] Murthy, R., Yang, L., Tan, J., Awalgaonkar, T. M., Zhou, Y., Heinecke, S., ... & Savarese, S. (2024). MobileAIBench: Benchmarking LLMs and LMMs for On-Device Use Cases. arXiv preprint arXiv:2406.10290.  
[b] Lu, Z., Li, X., Cai, D., Yi, R., Liu, F., Zhang, X., ... & Xu, M. (2024). Small Language Models: Survey, Measurements, and Insights. arXiv preprint arXiv:2409.15790.  
[c] Xu, J., Li, Z., Chen, W., Wang, Q., Gao, X., Cai, Q., & Ling, Z. (2024). On-device language models: A comprehensive review. arXiv preprint arXiv:2409.00088.  

### Presentation and Other Nitpicking

* Table 1, 2: Please put the captions above.
* Llama-3/3.1: Missing reference
* Throughput in Table 2 refers to generation throughput.
* MELTing: I believe the framework is called MELT.
* Figure 2: A boxplot presentation would be significantly richer in information, also showing peak memory usage (which can be the main factor for OOM errors).
* Figure 4: The overlay of the two workloads suggests that they are running simultaneously. An alternative visualization which also annotates what's happening at each time step would be more informational to the reader.
* §4.2: "To keep the models in suitable size [...] evaluate the total memory sage of models": I am not sure about what the authors mean here.
* §4.2: "[...] higher quantization escalates GPU memory [...]": Escalates might not be the correct word here.
soundness: 2
strengths: * The paper  quantifies the side-effects of quantization in various dimensions in language modelling on downstream tasks, including hallucinations and toxicity. This a valuable insight to the community.
* The multitude of devices that the authors have integrated are welcome, but lack the highest performing tier on Android (e.g. Snapdragon 8 Gen 2) and Linux (e.g. Jetson Orin AGX).
* I also greatly appreciate the integration of profiling measurements for reporting low-level CPU, GPU and memory utilization of the LLM inference workload.
summary: This paper describes the benchmarking results of several quantized Large Language Models on smartphones and edge devices. Specifically, it quantifies the CPU, GPU, memory and energy consumption of running inference on device, along with the accuracy and performance degradation across various dimensions (hallucinations, toxicity) as a consequence of quantization.
weaknesses: * The novel contributions of this paper are significantly fewer than originally stated and have been documented in prior work(s). Such include the evaluation of performance, energy and accuracy degradation of compressed LLMs on various devices, across similar frameworks. While I do agree that the downstream task performance quantification, along with the hallucination/alignment dimension is an important one, it seems to be the main novel contribution.
* The energy measurement methodology followed by the paper is largely software-based and thus depends on the vendor's implementation. Moreover, comparisons across ecosystems can be quite deceptive.
* The paper would benefit from a more rounded background/related work section, which describes prior work more faithfully and includes background information about the models and evaluated quantization methods.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: I'm not familiar with the 0-bit Llama model you mention on page 6. Isn't the bit count referring to quantization bits? How can that be zero? Is it a typo or just my ignorance?

Why FLIR in addition to power? There is a strong, deterministic relationship between power and temperature so the FLIR data should be useful primarily when spatial heterogeneity is important.  Is this for user discomfort due to contact with the phone or some other purpose? What was the approximate thermal time constant for the system spanning power dissipation location (mostly transistors) to smartphone surface? Did the benchmarks run long enough for the temperature distribution to reach steady state?

Why did higher bit width models often have lower quality measures than similar lower bit width models? Is this noise or something fundamental do to the influence of bit width on inductive biases or learning dynamics?
soundness: 4
strengths: This paper is in a good area. The efficient execution of LLMs on edge devices is important, both due to practical impact and because these platforms have tight resource constraints that can motivate new ideas that are more broadly applicable.

The "comprehensive" in the title is accurate regarding the range of measures and will set a high bar for others characterizing LLMs running on mobile and edge devices.

The paper has good tutorial value. Even though the focus is on benchmarking, it tersely and clearly describes several approaches to making LLMs more efficient.

The benchmarking findings are likely to be of interest to designers and researchers working on using LLMs on resource-constrained platforms.
summary: This paper describes a benchmarking process for compressed LLMs on mobile devices. It characterizes GPU and CPU utilization, speed, latency, memory requirements, power consumption, temperature, accuracies, and rates of undesirable behavior for several models on mobile and edge devices.
weaknesses: The paper has several findings regarding relative performance of different compression/quantization schemes. Some of these are counter-intuitive. The paper points these out but does not explain them. This holds true for findings on pages 9 and 10. The paper would probably be more interesting if it went deeper into the reasons for the surprising findings.

Benchmarking papers are generally not as interesting (I am thinking about conference attendees, not reviewers) as papers with novel design approaches or surprising and important scientific findings, and that holds true for this paper in my opinion. However, benchmarking is important so I view this as a small weakness. This work needs to be done and the authors did it well.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: What does complentary mean in Section 3.3.7: two complementary devices ?
I didn't get this statement in Sect. 4.2. CPU and GPU Utilization. Additionally, the iPhones exhibited ..., indicating the potential for optimization .... I though lower utilization is a good thing. In that case, why more optimization?
I wonder whether the inconsistencies, such as in Sect. 4.7, 3 bit is worse than 2 bit GWQ and 4-bit, is due to unrepeated experiments.
soundness: 3
strengths: The authors considers a large collection of LLMs, mobile platforms, and quatization schemes. The findings can be helpful for future researchers.
The organization of the paper is logical, which makes it easy to follow the author's arugments. 
The authors give some valuable insights based on the experimental results, such as the usefulness of 2-bit quantization for certain tasks in Section 4.5.
summary: The paper conducted extensive measurements of various on-device metrics and accuracy metrics for several popular LLMs on mobile devices to evaluate the effects of model size, quantization, hardware, inference engine, etc.
weaknesses: Given it was a heavily empirical paper, the authors did not clearly state whether the experiments were repeated multiple times to validate the results, or better, verified on multiple devices of the same family. It could be a challenge to gather all the devices, but the experimental results can be strengthened if the authors clearly stated that the devices were factory reset and repeated several times.
Even though the large collections of experiment is comprensive, it can be overwhelming to make sense of the results, especially without the author's interpretation. For example, Sect. 4.2. It is probably more meaningful to emphasize on some comparisons and highlight the difference and take-aways. 
Similar comments apply to Section 4.4. Any insights why 5-bit and 3-bit quantziation are worse?
Even though the authors claim it to be a framework, it is not readily reproducible by other labs, hence hard to be benchmarked by external teams.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: Thank you for submitting to ICLR 2025! 
I enjoy reading the paper. 
Helping developers better understand the tradeoff between model and system metrics for deploying LLMs on mobile devices is an important task.
I have a few comments and questions about the paper and it would be great if the authors could address them.

First, I think the paper should explain how automated testing or profiling of LLMs on the mobile device is conducted end-to-end at a high level.
The current Section 3 explains different metrics, LLMs, inference engines and so on in a quite detailed fashion.
However, it is not clear how a user or developer could use the benchmark to perform profiling.
It is also unclear how the framework could be extended to profile new mobile devices or new LLMs.
Also, do users need external equipment to fully use the benchmark?
For example, in Section 3.3.7, the paper mentions that thermal behavior analysis is done using a FLIR C5 thermal imaging camera and a professional USB power meter.

Despite a comprehensive analysis presented in the evaluation, I feel some of them sound trivial.
For example, for memory utilization, higher quantization levels consume more memory, while lower quantization than 4-bit reduces memory needs.
For CPU and GPU utilization, models with 4-bit quantization utilize more GPU duty cycles than those with 3-bit quantization.
Could the user get more insights from the results presented by the benchmark?
For example, for memory utilization, other than model weight, how much additional memory overhead is present in each setting and what may be the reasons?
For CPU and GPU utilization, could we obtain detailed layer-by-layer profiles and analyze what is the bottleneck operation?
What are some important implications or design decisions we can make from utilizing the benchmark?

Other questions:
1. In Section 4.2, it says "LLM inference is inherently memory-bound, and its memory utilization can be reduced significantly by memory bandwidth quantization". What is memory bandwidth quantization?
2. Can we evaluate the LLMs on mobile devices using CPU-only mode, or does the framework require the devices to have GPUs?
3. Do you have evaluation numbers for prefill and decode throughput respectively?
4. What is the batch size used in different settings when evaluating the throughput of LLMs?
5. How is the exact match and F1 score calculated when comparing quantized and non-quantized models? How to determine if the score is good or bad?
6. In Section 4.4, it says "Interestingly, the 5-bit and 3-bit models underperformed slightly. K-quant (ggml) 3-bit model produced more hallucinations and toxic content than the 2-bit model." How are hallucination and toxicity observed in Figure 8?
soundness: 3
strengths: 1. The paper is written in good quality and easy to follow.
2. The framework supports a range of devices and LLMs.
3. The experiments are presented in detail with an analysis of the key metrics supported by the framework.
summary: Serving LLMs on mobile devices is becoming increasingly important due to privacy and security requirements.
Deploying LLMs on mobile platforms is challenging as the developer may need to trade-off between model quality, inference performance and power efficiency.
Current LLM benchmarks primarily target cloud platforms, with only a few for limited mobile devices.
This paper proposes PalmBench, a comprehensive benchmarking framework to evaluate the user experience of LLMs on mobile devices.
The framework evaluates LLM on metrics including efficiency, accuracy and toxicity.
The framework supports multiple quantized LLMs and multiple mobile devices with different types of OSs.
weaknesses: 1. Lack of explanation on how the automated testing is performed end-to-end.
2. Unclear how easy it is to support new mobile devices or new LLMs.
3. Some of the analysis and results seem to be trivial.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: - What's the control flow (hierarchy) of the framework and how do different components interact? For instance, the paper lacks crucial details about how user queries are processed and evaluated using their framework, particularly in relation to AGI

- The Fig.1 shows a linear flow from models to automation framework to metrics, but how do different quantization methods (Group-wise, GGML, GPTQ, AWQ) integrate into this flow? Shouldn't the quantization block be positioned between Models and Frameworks since models need to be quantized before being deployed through MLC or llama.cpp?

- I was trying to reproduce the supplement materials, and trying to perform some benchmark tasks. However, I am unable to reproduce the experiments shown in the paper. I believe the supplementary materials is not complete.  It looks like it's based on the NVTOP and processes the returning string. I'm trying to perform some benchmarking, could you please provide some details how to run your pipeline, ?

- Could you provide a detailed description showing how AGI, custom apps, and LLM frameworks interact in your automated system? (e.g.,What's the exact interface between your custom apps and the LLM frameworks (MLC/llama.cpp)?

- How you process user's query to evaluate the model? should it must have llama.cpp support, what if my model is not in llama.cpp model zoo?
soundness: 2
strengths: - A benchmark for evaluating LLM for edge computing scenarios is necessary, and this study focuses on this.
- Experimenting covering iOS, Android, and edge devices with detailed hardware specifications
summary: This paper introduces PalmBench, a benchmark framework for evaluating LLM on edge computing scenarios, and also conducted evaluation on various models under different benchmark datasets. 
The key goal of this paper is to complement the lacking of an LLM benchmark at the edge.
The proposed framework allows automated testing of various LLMs with different compression settings across multiple mobile devices.  This paper also evaluates various popular LLMs with different benchmarks.
weaknesses: - The overall system design of PalmBench is vague to me and not elaborate in the paper; I am not able to get information on how each of the components collaborates in Figure 1 and what's the hierarchy of the system. 

- The Benchmark Automation Implementation details are lacking, it's hard to know how PalmBench to process different hardware platform benchmark query.

- There is already have Benchmark [1] for evaluating LLM on Edge-Device, which should be properly discussed in this paper.


[1] https://github.com/TianjinYellow/EdgeDeviceLLMCompetition-Starting-Kit?tab=readme-ov-file#submission-requirements

Meta-review: The paper presents PalmBench, a benchmarking framework designed to assess the performance of compressed Large Language Models (LLMs) on mobile devices. It offers a comprehensive evaluation of LLMs across various quantization configurations and mobile platforms with diverse hardware capabilities. The study emphasizes resource efficiency, including memory and power consumption, as well as the quality of output, particularly focusing on the frequency of hallucinations and toxic content.

PalmBench introduces a novel benchmarking framework that addresses a significant gap in the evaluation of LLMs on mobile devices. The paper covers a wide range of LLMs and quantization schemes, providing valuable insights into their performance on mobile platforms.

Initially, reviewers expressed concerns about the lack of clarity in the system design of PalmBench and the insufficient comparison with existing benchmarks. During the rebuttal period, reviewers emphasized the need for more detailed real-world deployment data and clearer explanations of the system design. The authors responded effectively by committing to open-source their framework and providing additional details, which clarified how PalmBench integrates different quantization methods and mobile platforms. They also expanded the comparison with existing benchmarks.

As the Area Chair, I have carefully reviewed the feedback from the reviewers and the authors' responses. The initial concern regarding the lack of real-world deployment data was addressed by the authors’ commitment to provide more comprehensive data in future work, acknowledging the current limitations of their study. The vagueness in the system design was clarified with additional explanations and diagrams, improving the understandability of PalmBench's architecture and its components' interactions. Although integrating different quantization methods into the benchmarking process posed practical challenges on hardware, the authors’ rebuttal and subsequent revisions alleviated these concerns. They demonstrated the feasibility and practicality of their approach through clear explanations and a roadmap for future integrations.

The authors have made significant efforts to address the initial concerns. Their proactive engagement with the reviewers' feedback and the substantial improvements made to the paper are commendable. Given the competitive nature of the conference, the final recommendation, after thorough consideration, is to accept this submission. However, I encourage the authors to consider further incorporating the reviewers' suggestions in the final version.

Decision: Accept (Poster)