PALMBENCH: A COMPREHENSIVE BENCHMARK OF COMPRESSED LARGE LANGUAGE MODELS ON MOBILE PLATFORMS

Authors: Yilong Li, Jingyu Liu, Hao Zhang, M Badri Narayanan, Utkarsh Sharma, Shuai Zhang, Yijing Zeng, Jayaram Raghuram, Suman Banerjee
Primary area: datasets and benchmarks
Keywords: Mobile Platforms, Large Language Models, Quantization, Benchmark

Abstract:
Deploying large language models (LLMs) locally on mobile devices is advantageous in scenarios where transmitting data to remote cloud servers is either undesirable due to privacy concerns or impractical due to network connection. Recent advancements have facilitated the local deployment of LLMs. However, local deployment also presents challenges, particularly in balancing quality (generative performance), latency, and throughput within the hardware constraints of mobile devices. In this paper, we introduce our lightweight, all-in-one automated benchmarking framework that allows users to evaluate LLMs on mobile devices. We provide a comprehensive benchmark of various popular LLMs with different quantization configurations (both weights and activations) across multiple mobile platforms with varying hardware capabilities. Unlike traditional benchmarks that assess full-scale models on high-end GPU clusters, we focus on evaluating resource efficiency (memory and power consumption) and harmful output for compressed models on mobile devices. Our key observations include: i) differences in energy efficiency and throughput across mobile platforms; ii) the impact of quantization on memory usage, GPU execution time, and power consumption; and iii) accuracy and performance degradation of quantized models compared to their non-quantized counterparts; and iv) the frequency of hallucinations and toxic content generated by compressed LLMs on
mobile devices.

Decision: accept (poster)
Reviewer scores: 5, 8, 5, 6, 5

Reviews and discussion:
<thread>
creation date:
1730606965402

thread_type:
official_review

<thread_object>
creator: reviewer_6lsd

<content>
summary:
This paper introduces PalmBench, a benchmark framework for evaluating LLM on edge computing scenarios, and also conducted evaluation on various models under different benchmark datasets. 
The key goal of this paper is to complement the lacking of an LLM benchmark at the edge.
The proposed framework allows automated testing of various LLMs with different compression settings across multiple mobile devices.  This paper also evaluates various popular LLMs with different benchmarks.

strengths:
- A benchmark for evaluating LLM for edge computing scenarios is necessary, and this study focuses on this.
- Experimenting covering iOS, Android, and edge devices with detailed hardware specifications

weaknesses:
- The overall system design of PalmBench is vague to me and not elaborate in the paper; I am not able to get information on how each of the components collaborates in Figure 1 and what's the hierarchy of the system. 

- The Benchmark Automation Implementation details are lacking, it's hard to know how PalmBench to process different hardware platform benchmark query.

- There is already have Benchmark [1] for evaluating LLM on Edge-Device, which should be properly discussed in this paper.


[1] https://github.com/TianjinYellow/EdgeDeviceLLMCompetition-Starting-Kit?tab=readme-ov-file#submission-requirements

questions:
- What's the control flow (hierarchy) of the framework and how do different components interact? For instance, the paper lacks crucial details about how user queries are processed and evaluated using their framework, particularly in relation to AGI

- The Fig.1 shows a linear flow from models to automation framework to metrics, but how do different quantization methods (Group-wise, GGML, GPTQ, AWQ) integrate into this flow? Shouldn't the quantization block be positioned between Models and Frameworks since models need to be quantized before being deployed through MLC or llama.cpp?

- I was trying to reproduce the supplement materials, and trying to perform some benchmark tasks. However, I am unable to reproduce the experiments shown in the paper. I believe the supplementary materials is not complete.  It looks like it's based on the NVTOP and processes the returning string. I'm trying to perform some benchmarking, could you please provide some details how to run your pipeline, ?

- Could you provide a detailed description showing how AGI, custom apps, and LLM frameworks interact in your automated system? (e.g.,What's the exact interface between your custom apps and the LLM frameworks (MLC/llama.cpp)?

- How you process user's query to evaluate the model? should it must have llama.cpp support, what if my model is not in llama.cpp model zoo?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730622705291

thread_type:
official_review

<thread_object>
creator: reviewer_w3ys

<content>
summary:
Serving LLMs on mobile devices is becoming increasingly important due to privacy and security requirements.
Deploying LLMs on mobile platforms is challenging as the developer may need to trade-off between model quality, inference performance and power efficiency.
Current LLM benchmarks primarily target cloud platforms, with only a few for limited mobile devices.
This paper proposes PalmBench, a comprehensive benchmarking framework to evaluate the user experience of LLMs on mobile devices.
The framework evaluates LLM on metrics including efficiency, accuracy and toxicity.
The framework supports multiple quantized LLMs and multiple mobile devices with different types of OSs.

strengths:
1. The paper is written in good quality and easy to follow.
2. The framework supports a range of devices and LLMs.
3. The experiments are presented in detail with an analysis of the key metrics supported by the framework.

weaknesses:
1. Lack of explanation on how the automated testing is performed end-to-end.
2. Unclear how easy it is to support new mobile devices or new LLMs.
3. Some of the analysis and results seem to be trivial.

questions:
Thank you for submitting to ICLR 2025! 
I enjoy reading the paper. 
Helping developers better understand the tradeoff between model and system metrics for deploying LLMs on mobile devices is an important task.
I have a few comments and questions about the paper and it would be great if the authors could address them.

First, I think the paper should explain how automated testing or profiling of LLMs on the mobile device is conducted end-to-end at a high level.
The current Section 3 explains different metrics, LLMs, inference engines and so on in a quite detailed fashion.
However, it is not clear how a user or developer could use the benchmark to perform profiling.
It is also unclear how the framework could be extended to profile new mobile devices or new LLMs.
Also, do users need external equipment to fully use the benchmark?
For example, in Section 3.3.7, the paper mentions that thermal behavior analysis is done using a FLIR C5 thermal imaging camera and a professional USB power meter.

Despite a comprehensive analysis presented in the evaluation, I feel some of them sound trivial.
For example, for memory utilization, higher quantization levels consume more memory, while lower quantization than 4-bit reduces memory needs.
For CPU and GPU utilization, models with 4-bit quantization utilize more GPU duty cycles than those with 3-bit quantization.
Could the user get more insights from the results presented by the benchmark?
For example, for memory utilization, other than model weight, how much additional memory overhead is present in each setting and what may be the reasons?
For CPU and GPU utilization, could we obtain detailed layer-by-layer profiles and analyze what is the bottleneck operation?
What are some important implications or design decisions we can make from utilizing the benchmark?

Other questions:
1. In Section 4.2, it says "LLM inference is inherently memory-bound, and its memory utilization can be reduced significantly by memory bandwidth quantization". What is memory bandwidth quantization?
2. Can we evaluate the LLMs on mobile devices using CPU-only mode, or does the framework require the devices to have GPUs?
3. Do you have evaluation numbers for prefill and decode throughput respectively?
4. What is the batch size used in different settings when evaluating the throughput of LLMs?
5. How is the exact match and F1 score calculated when comparing quantized and non-quantized models? How to determine if the score is good or bad?
6. In Section 4.4, it says "Interestingly, the 5-bit and 3-bit models underperformed slightly. K-quant (ggml) 3-bit model produced more hallucinations and toxic content than the 2-bit model." How are hallucination and toxicity observed in Figure 8?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730650203991

thread_type:
official_review

<thread_object>
creator: reviewer_px3e

<content>
summary:
The paper conducted extensive measurements of various on-device metrics and accuracy metrics for several popular LLMs on mobile devices to evaluate the effects of model size, quantization, hardware, inference engine, etc.

strengths:
The authors considers a large collection of LLMs, mobile platforms, and quatization schemes. The findings can be helpful for future researchers.
The organization of the paper is logical, which makes it easy to follow the author's arugments. 
The authors give some valuable insights based on the experimental results, such as the usefulness of 2-bit quantization for certain tasks in Section 4.5.

weaknesses:
Given it was a heavily empirical paper, the authors did not clearly state whether the experiments were repeated multiple times to validate the results, or better, verified on multiple devices of the same family. It could be a challenge to gather all the devices, but the experimental results can be strengthened if the authors clearly stated that the devices were factory reset and repeated several times.
Even though the large collections of experiment is comprensive, it can be overwhelming to make sense of the results, especially without the author's interpretation. For example, Sect. 4.2. It is probably more meaningful to emphasize on some comparisons and highlight the difference and take-aways. 
Similar comments apply to Section 4.4. Any insights why 5-bit and 3-bit quantziation are worse?
Even though the authors claim it to be a framework, it is not readily reproducible by other labs, hence hard to be benchmarked by external teams.

questions:
What does complentary mean in Section 3.3.7: two complementary devices ?
I didn't get this statement in Sect. 4.2. CPU and GPU Utilization. Additionally, the iPhones exhibited ..., indicating the potential for optimization .... I though lower utilization is a good thing. In that case, why more optimization?
I wonder whether the inconsistencies, such as in Sect. 4.7, 3 bit is worse than 2 bit GWQ and 4-bit, is due to unrepeated experiments.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730673649366

thread_type:
official_review

<thread_object>
creator: reviewer_em3f

<content>
summary:
This paper describes a benchmarking process for compressed LLMs on mobile devices. It characterizes GPU and CPU utilization, speed, latency, memory requirements, power consumption, temperature, accuracies, and rates of undesirable behavior for several models on mobile and edge devices.

strengths:
This paper is in a good area. The efficient execution of LLMs on edge devices is important, both due to practical impact and because these platforms have tight resource constraints that can motivate new ideas that are more broadly applicable.

The "comprehensive" in the title is accurate regarding the range of measures and will set a high bar for others characterizing LLMs running on mobile and edge devices.

The paper has good tutorial value. Even though the focus is on benchmarking, it tersely and clearly describes several approaches to making LLMs more efficient.

The benchmarking findings are likely to be of interest to designers and researchers working on using LLMs on resource-constrained platforms.

weaknesses:
The paper has several findings regarding relative performance of different compression/quantization schemes. Some of these are counter-intuitive. The paper points these out but does not explain them. This holds true for findings on pages 9 and 10. The paper would probably be more interesting if it went deeper into the reasons for the surprising findings.

Benchmarking papers are generally not as interesting (I am thinking about conference attendees, not reviewers) as papers with novel design approaches or surprising and important scientific findings, and that holds true for this paper in my opinion. However, benchmarking is important so I view this as a small weakness. This work needs to be done and the authors did it well.

questions:
I'm not familiar with the 0-bit Llama model you mention on page 6. Isn't the bit count referring to quantization bits? How can that be zero? Is it a typo or just my ignorance?

Why FLIR in addition to power? There is a strong, deterministic relationship between power and temperature so the FLIR data should be useful primarily when spatial heterogeneity is important.  Is this for user discomfort due to contact with the phone or some other purpose? What was the approximate thermal time constant for the system spanning power dissipation location (mostly transistors) to smartphone surface? Did the benchmarks run long enough for the temperature distribution to reach steady state?

Why did higher bit width models often have lower quality measures than similar lower bit width models? Is this noise or something fundamental do to the influence of bit width on inductive biases or learning dynamics?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730674121451

thread_type:
official_review

<thread_object>
creator: reviewer_1kjr

<content>
summary:
This paper describes the benchmarking results of several quantized Large Language Models on smartphones and edge devices. Specifically, it quantifies the CPU, GPU, memory and energy consumption of running inference on device, along with the accuracy and performance degradation across various dimensions (hallucinations, toxicity) as a consequence of quantization.

strengths:
* The paper  quantifies the side-effects of quantization in various dimensions in language modelling on downstream tasks, including hallucinations and toxicity. This a valuable insight to the community.
* The multitude of devices that the authors have integrated are welcome, but lack the highest performing tier on Android (e.g. Snapdragon 8 Gen 2) and Linux (e.g. Jetson Orin AGX).
* I also greatly appreciate the integration of profiling measurements for reporting low-level CPU, GPU and memory utilization of the LLM inference workload.

weaknesses:
* The novel contributions of this paper are significantly fewer than originally stated and have been documented in prior work(s). Such include the evaluation of performance, energy and accuracy degradation of compressed LLMs on various devices, across similar frameworks. While I do agree that the downstream task performance quantification, along with the hallucination/alignment dimension is an important one, it seems to be the main novel contribution.
* The energy measurement methodology followed by the paper is largely software-based and thus depends on the vendor's implementation. Moreover, comparisons across ecosystems can be quite deceptive.
* The paper would benefit from a more rounded background/related work section, which describes prior work more faithfully and includes background information about the models and evaluated quantization methods.

questions:
### Motivation

* The authors suggest in the introduction that "running LLMs locally" can lead to "increased reliability". Personally and from experience, I would not be that confident on making such claims, i.e. that cloud deployments are less reliable that deploying locally, especially given the infancy of the evaluated frameworks. I would appreciate some more substantiation here from the authors.

### Methodology

* Wrt methodology and experimental setup, the paper misses substantial information that hurt the overall reproducibility potential. Such omissions include:
    - Operating system and framework versions
    - Automation of iOS interface
    - How different components in Figure 1 actually coordinate and communicate.
* Would the authors be willing to open-source their benchmarking system?
* It is unclear whether the authors have converted the models themselves or have used the pre-compiled versions offered in GGUF and MLC repositories.
* How do the authors ensure that the profiling methodology does not interfere with the behavior of the experiment? Also, how do the authors isolate the power lanes from the USB communication?
* The authors claim that they are using a "professional USB power meter for accurate power consumption measurements". However, it is not clear how this yields the information needed, as devices are battery powered and not USB-powered. As such, the power draw from the USB does not yield any information related to energy consumption from a workload on device.

### Evaluation

* The evaluation does not integrate variance metrics. Should it be assumed that the experiments have run once?
* What is the decoding strategy and hyperparameters that the authors use during evaluation?
* With respect to temperature, does the average represent "average over time" or "average over surface"?
* A missing dimension that is worth exploring in such a paper is the tradeoff between size due to parameters and due to quantization (or another compression method). For example, is it better using a lower-bitwidth llama-3.1-8b model on 4bits or a llama-3.2-3b model on 8 bits?
* What is the bitwidth used in Figure 2 across models?
* In Figure 3, the GPU utilization across phones is quite large, which comes in contrast with the "memory bounded" workload claim of the authors. I would appreciate some more insights here.
* §4.2: "3-bit quantization results in lower CPU and GPU usage [...] decreased data transfers [...] reduced inference workload": I am not sure this claim is correct or substantiated properly.
* A great source of information to the reader, also for showcasing the memory-boundedness of the workload would be to plot a roofline model for devices/models.
* The authors claim that iPhones provide significantly higher throughputs compared to other devices. This may not be painting the whole picture, as it is not clear whether this throughput can be sustained compared to the actively cooled Jetson Nano for instance.

### Related work

* The authors claim that prior work (MELT) has examined resource utilization and energy efficiency in a limited manner, and did not explore GPU workloads. However, this is not true. 
    * Additionally, the authors do not run llama.cpp on iOS, which prior work has done.
    * Palmbench does not measure power consumption via hardware probes on phones and do not measure it at all on edge devices.
    * Palmbench does not report on prefill rates, which prior work does.
    * Palmbench does not integrate high-end edge devices (e.g. Jetson AGX) and different power profiles, which prior work does.
* Moreover, the authors have unfortunately missed other related work in the domain of on-device LLM benchmarking [a-c].

[a] Murthy, R., Yang, L., Tan, J., Awalgaonkar, T. M., Zhou, Y., Heinecke, S., ... & Savarese, S. (2024). MobileAIBench: Benchmarking LLMs and LMMs for On-Device Use Cases. arXiv preprint arXiv:2406.10290.  
[b] Lu, Z., Li, X., Cai, D., Yi, R., Liu, F., Zhang, X., ... & Xu, M. (2024). Small Language Models: Survey, Measurements, and Insights. arXiv preprint arXiv:2409.15790.  
[c] Xu, J., Li, Z., Chen, W., Wang, Q., Gao, X., Cai, Q., & Ling, Z. (2024). On-device language models: A comprehensive review. arXiv preprint arXiv:2409.00088.  

### Presentation and Other Nitpicking

* Table 1, 2: Please put the captions above.
* Llama-3/3.1: Missing reference
* Throughput in Table 2 refers to generation throughput.
* MELTing: I believe the framework is called MELT.
* Figure 2: A boxplot presentation would be significantly richer in information, also showing peak memory usage (which can be the main factor for OOM errors).
* Figure 4: The overlay of the two workloads suggests that they are running simultaneously. An alternative visualization which also annotates what's happening at each time step would be more informational to the reader.
* §4.2: "To keep the models in suitable size [...] evaluate the total memory sage of models": I am not sure about what the authors mean here.
* §4.2: "[...] higher quantization escalates GPU memory [...]": Escalates might not be the correct word here.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732309758143

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Thanks for all revewers' feedback

comment:
We sincerely thank all the reviewers for their valuable comments and thoughtful feedback. We apologize if our response and revisions came later than expected; we have carefully and thoroughly revised the manuscript in light of your insightful suggestions. Below, we address some of the common questions and comments:

*
__Open-source and Reproducibility:__ We have uploaded some datasets and module testing scripts as supplemental materials. Additionally, we plan to open source all the testing scripts and frameworks, including firmware for edge devices, mobile apps for iPhones, and AGI used on Android devices. Due to the upload size limitations, the firmware and mobile app project files could not be included as supplemental materials, but we will make all these files publicly available to ensure the reproducibility of our experiments.

*
__Lack of system details:__ We noticed that the existing description lacked system design details, particularly regarding how our framework performs benchmarks in an end-to-end manner. To address this, we have revised Figure 1 with workflow and included additional details in __Blue color__. We invite you to review the updated draft. 

*
__Experiment Repetition Details:__  In the revised version, we have added these details. All experiments were conducted 10 times to ensure consistency, with identical settings for both frameworks, llama.cpp and MLC, including temperature and sampling hyperparameters.
We added one more section in the Appendix for framework hyperparameters.

*
CPU utilization, GPU utilization, and memory usage are reported as average values rather than with error bars or box plots because, on a single device running a specific model, resource usage—particularly memory usage—shows minimal fluctuation. This holds true even with varying datasets and prompt inputs, making visualizations like error bars less meaningful and less intuitive for comparison.
*
__Deep Analysis and layer-by-layer profiling:__  To identify bottlenecks in running LLMs on mobile and edge devices, we analyzed memory breakdown and layer-by-layer GPU utilization. The memory breakdown highlights the contributions of weights, KV cache, activations, and overhead, influenced by quantization bitwidth and algorithms. GPU profiling across two transformer blocks reveals that self-attention layers achieve over 95% GPU utilization, potentially slowing inference, while FFN layers use significantly less. These insights guide optimizations to enhance model performance. To be noted, the data we measured comes from Google's official AGI tool and MLC's internal status printing messages(the newest provides detailed information, such as KV cache memory and overhead by default). We believe these are among the most reliable and accurate data sources available.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732595440936

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Explanation of Power Consumption Measurement

comment:
To explain the accuracy of power consumption data obtained through power rails or the __On-Device Power Monitor (ODPM)__ from official profiling tools.  Let me clarify how our power consumption data of mobile phones is measured via power rails, using the example of the Android Pixel 7, which employs (On-Device Power Monitor) ODPM.

Unfortunately I cannot include figures here, I would be happy to provide links to the official Google Android documentation on ODPM and the datasheet for the PMU used in these devices[2]. These references will further validate the accuracy of our power measurement method [1][2][3][4][5]. All of these references are technical documents from official websites.

References [3][4][5] introduce the Power Profiler from the Android Development Kit, which leverages ODPM to obtain precise power consumption data.
>ODPM measures power consumption at the device level—not specific to any app. 
>Power Profiler reads power consumption data from the ODPM, which is only available on Pixel 6 and subsequent Pixel devices running Android 10 (API level 29) and higher.

ODPM is essential for accurate power consumption measurement under battery power, as detailed in the official Android documentation on power rails and charging[4]. When connected to a USB cable, devices measure charge flow into the battery (i.e., charging), leading to positive current readings and increased total charge. This distorts power data, reflecting charging activity rather than actual GPU power usage. Moreover, USB connections often trigger high-performance modes, unlike the low-power modes typically used on battery. As a result, USB-based measurements are unsuitable for accurately analyzing LLM inference on GPUs.

As described in the document:
> Battery counters measure the charge flowing in and out of the battery. If the device is plugged to a USB cable, you will likely observe a positive instantaneous current and an increase of the total charge, denoting the fact that charge is flowing in the battery (i.e. charging it) rather than out.

>This can make measurements in lab settings problematic.

Why ODPM is more accurate?

>Recent version of Android introduced the support for more advanced power monitoring at the hardware subsystem level, known as “On-Device Power Rail Monitors” (ODPMs). These counters measure the energy drained by (groups of) hardware units.
Unlike the battery counters, they are not affected by the charging/discharging state of the battery, because they measure power downstream of the battery.

ODPM differs from battery counters as it directly uses on device __hardware units__ like the Power Management Unit (PMU) to measure power consumption. Reference [4] specifically mentions the __power monitoring chip__, which refers to the PMU.

The PMU provides direct and accurate battery power measurements, which external cabled tools cannot replace[2][3][4]. Connecting external devices can alter the phone's power management state or change its power supply behavior, leading to inaccurate results, as indicated in [3].

We use AGI [1] as it offers a more advanced power profiling capability compared to Android Studio, fully leveraging the specifications of ODPM for precise measurements.

Here I provide one widely used PMU[2] datasheet.

[1]Android GPU Inspector (AGI): https://developer.android.com/agi

[2]The datasheet of the PMU used on the device: https://www.nxp.com/docs/en/application-note/AN4199.pdf

[3]Android Power Profiler: https://developer.android.com/studio/profile/power-profiler

[4]Google OPDM: https://android.googlesource.com/platform//external/perfetto/+/b4a6d11ec1c12f391c3f9dab2be4f3b5152994f2/docs/data-sources/battery-counters.md

[5] Android Power Monitor: https://developer.android.com/reference/android/os/PowerMonitor
</content>
</thread_object>

</thread>
<thread>
creation date:
1732685697913

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Additional Examples of Irrelevant, Arbitrary Outputs and Hallucinations.

comment:
Our evaluation indicates that 5-bit and 3-bit quantization are more likely to produce irrelevant, arbitrary outputs and hallucinations. This behavior is attributed to the limitations of the framework and the quantization algorithms. Below are some examples of incorrect outputs generated by 5-bit quantization:

__Hallucination__

Gemma-3B-q5 llama.cpp
> User: Who was the 45th president of the United States, and what were their main achievements in ancient Greece?

> AI:The 45th president of the United States was Abraham Lincoln, who was also known for his contributions to ancient Greek philosophy, particularly in mentoring Socrates and introducing democratic reforms in Athens.

__Irrelevant Outputs__

Llama3-8B-q5 llama.cpp LLMFarm (llama.cpp)
> Users: Hi
> AI: 
> <end_of_turn>model <start_of_turn>print print("hello world") <end_of_turn> <start_of_turn>with open() <end_of_turn>with open('example.txt', 'w') as f: f.write('goodbye') f.write('goodbye') <end_of_turn> <start_of_turn>print("goodbye world")<end_of_turn> <start_of_turn>print(f"goodbye {i}")<end_of_turn> <end_of_turn> <start_of_turn>for i in range(1,10): <end_of_turn> print(f"goodbye {i}") <end_of_turn> <start_of_turn>for i in range(1,10): print(f"goodbye {i}"<end_of_turn> <end_of_turn> <start_of_turn>with open('example.txt', 'r') as f: <end_of_turn>print(f.read())<end_of_turn> <start_of_turn>print(f.read() + "\n")<end_of_turn> <end_of_turn> <start_of_turn>wget url="https://www.python.org/": <end_of_turn> print(wget)<end_of_turn> <start_of_turn>print(wget) <end_of_turn>...<end_of_turn> anagrams[char] = anagrams.get(char, 0) + 1 return anagrams <end_of_turn> <end_of_turn> <start_of_turn>def is_prime(n): <end_of_turn> if n <= 1: <end_of_turn> return False <end_of_turn> <end_of_turn> <start_of_turn>def get_prime_factors(n): <end_of_turn> factors = [] <end_of_turn> i = 2 <end_of_turn> while n > 1: <end_of_turn> if n % i == 0: <end_of_turn> factors.append(i) <end_of_turn> while i * i <= n n n n n n n n n n n n n n n n n …..n n+zone. in a n^2 some and<|Fun's the, and a with a| and I the. |n the a, the, n the, n't a, isill is the and the the be and., is is C_LIMIT  isable is the, the the+ the, is and+ is+ a it the the is the the the is the the the to the the this'< the the the is is the is the of of the a it| the, a the the to the the is the, a is the is the is the the is the  the is the is++ the+ thely to and the the is is andly that. the the< isore the it
</content>
</thread_object>

</thread>
<thread>
creation date:
1740902364712

thread_type:
public_comment

<thread_object>
creator: ~yilong_li2

<content>
title:
Camera-Ready Version Submission and Updates

comment:
Dear AC Chair and Reviewers,

I hope you are doing well. We have just uploaded the camera-ready version of our paper, incorporating the revisions prompted by your feedback. Specifically, we addressed the concerns raised in your reviews by:

1. **Clarifying Confusing Descriptions**  
   We rephrased and refined several sections to ensure our explanations are more precise and more accurate.  
2. **Adding Further Details**  
   We expanded on the methodology and experimental setup where necessary, providing additional evidence and context for our findings.  
3. **New Experiment on Deepseek-R1**  
   We included a comprehensive evaluation using Deepseek R1—currently one of the most widely discussed models—to further strengthen our empirical results.

We greatly appreciate your constructive comments and suggestions, as they have helped us significantly improve the paper’s overall clarity and quality. If there are any further questions or additional feedback, please let us know.

Thank you again for your time and assistance throughout the review process.

Best regards,  Authors of paper #13109
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presents PalmBench, a benchmarking framework designed to assess the performance of compressed Large Language Models (LLMs) on mobile devices. It offers a comprehensive evaluation of LLMs across various quantization configurations and mobile platforms with diverse hardware capabilities. The study emphasizes resource efficiency, including memory and power consumption, as well as the quality of output, particularly focusing on the frequency of hallucinations and toxic content.

PalmBench introduces a novel benchmarking framework that addresses a significant gap in the evaluation of LLMs on mobile devices. The paper covers a wide range of LLMs and quantization schemes, providing valuable insights into their performance on mobile platforms.

Initially, reviewers expressed concerns about the lack of clarity in the system design of PalmBench and the insufficient comparison with existing benchmarks. During the rebuttal period, reviewers emphasized the need for more detailed real-world deployment data and clearer explanations of the system design. The authors responded effectively by committing to open-source their framework and providing additional details, which clarified how PalmBench integrates different quantization methods and mobile platforms. They also expanded the comparison with existing benchmarks.

As the Area Chair, I have carefully reviewed the feedback from the reviewers and the authors' responses. The initial concern regarding the lack of real-world deployment data was addressed by the authors’ commitment to provide more comprehensive data in future work, acknowledging the current limitations of their study. The vagueness in the system design was clarified with additional explanations and diagrams, improving the understandability of PalmBench's architecture and its components' interactions. Although integrating different quantization methods into the benchmarking process posed practical challenges on hardware, the authors’ rebuttal and subsequent revisions alleviated these concerns. They demonstrated the feasibility and practicality of their approach through clear explanations and a roadmap for future integrations.

The authors have made significant efforts to address the initial concerns. Their proactive engagement with the reviewers' feedback and the substantial improvements made to the paper are commendable. Given the competitive nature of the conference, the final recommendation, after thorough consideration, is to accept this submission. However, I encourage the authors to consider further incorporating the reviewers' suggestions in the final version.