PALMBENCH: A COMPREHENSIVE BENCHMARK OF COMPRESSED LARGE LANGUAGE MODELS ON MOBILE PLATFORMS
Authors: Yilong Li
Jingyu Liu
Hao Zhang
M Badri Narayanan
Utkarsh Sharma
Shuai Zhang
Yijing Zeng
Jayaram Raghuram
Suman Banerjee
Venue: ICLR 2025 Poster (2025)
Keywords: Mobile Platforms
Large Language Models
Quantization
Benchmark
Primary area: datasets and benchmarks
TL;DR: Benchmarking compressed Large Language Models on mobile devices

Abstract: Deploying large language models (LLMs) locally on mobile devices is advantageous in scenarios where transmitting data to remote cloud servers is either undesirable due to privacy concerns or impractical due to network connection. Recent advancements have facilitated the local deployment of LLMs. However, local deployment also presents challenges, particularly in balancing quality (generative performance), latency, and throughput within the hardware constraints of mobile devices. In this paper, we introduce our lightweight, all-in-one automated benchmarking framework that allows users to evaluate LLMs on mobile devices. We provide a comprehensive benchmark of various popular LLMs with different quantization configurations (both weights and activations) across multiple mobile platforms with varying hardware capabilities. Unlike traditional benchmarks that assess full-scale models on high-end GPU clusters, we focus on evaluating resource efficiency (memory and power consumption) and harmful output for compressed models on mobile devices. Our key observations include: i) differences in energy efficiency and throughput across mobile platforms; ii) the impact of quantization on memory usage, GPU execution time, and power consumption; and iii) accuracy and performance degradation of quantized models compared to their non-quantized counterparts; and iv) the frequency of hallucinations and toxic content generated by compressed LLMs on
mobile devices.

Published as a conference paper at ICLR 2025
PALMBENCH: A COMPREHENSIVE BENCHMARK OF COMPRESSED LARGE LANGUAGE MODELS ON MOBILE PLATFORMS
Yilong Li', Jingyu Liu', Hao Zhang', M Badri Narayanan' Utkarsh Sharma', Shuai Zhang?
Yijing Zeng". Jayaram Raghuram', Suman Banerjee*
University of Wisconsin Madison, Amazzo Web Services AI, USA
ABSTRACT
Deploying large language models (LLMs) locally mobile devices is advan- tageous in scenarios where transmitting data to remote cloud servers is either undesirable due privacy concerns impractical due to network connection. Recent advancements (MLC) 2023a Gerganov|
(202)) have facilitated the local deployment of LLMs. However, local deployment also presents challenges, par- ticularly in balancing quality (generative performance), latency, and throughput within the hardware constraints of mobile devices. In this paper. we introduce our lightweight, all-in-one automated benchmarking framework that allows users to evaluate LLMs on mobile devices. We provide a comprehensive benchmark of various popular LLMs with different quantization configurations (both weights and activations) across multiple mobile platforms with varying hardware capa- bilities. Unlike traditional benchmarks that assess full-scale models on high-end GPU clusters, we focus on evaluating resource efficiency (memory and power consumption) and harmful output for compressed models mobile devices. Our key observations include: i) differences in energy efficiency and throughput across mobile platforms; ii) the impact of quantization on memory usage, GPU execution time, and power consumption; and iii) accuracy and performance degradation of quantized models compared to their non-quantized counterparts; and iv) the frequency of hallucinations and toxic content generated by compressed LLMs on mobile devices
INTRODUCTION
Large Language Models (LLMs) such as ChatGPT OpenAl 2023), Claude (Anthropic) 2023).
and Llama TTovrro et al. 2023a b b Llamal 2024 are powertul generative models that are rev- olutionizing interactive communication and various natural language processing tasks, including guestion-answering. document summarization, abstract reasoning, and code auto-completion (e.g., Github Copilot (Github)). LLMs require significant computational and memory resources due their huge number of parameters (e.g., MT-NLG 530B (Smith et al., 2022),, ), making them more suitable for running on cloud infrastructures with high-end powerful GPU clusters. While significant attention has been dedicated to cloud based LLMs, there is growing need to run LLMs on resource-constraing mobile devices to obtain some key benefits. (1) Privacy and Security: Processing user data locally on mobile devices helps protect user privacy and enhances data security. There is also less risk of data breaches or unauthorized access to sensitive information. (2) No Cloud Reliance: By running LLMSs locally, mobile applications can reduce their dependence on cloud services for language processing tasks. This can lead to cost savings and increased reliability, as the application's functionality not reliant on the availability and performance of remote servers (Hu et al. 2024 (3) Offline Access: By running LLMs on mobile devices, users can access powerful language processing capabilities even when they not connected (or have unreliable connection) to the Internet.
The rapidly flourishing LLM ecosystem, including various large models, architectures and frame- works, presents both opportunities and challenges for developers and researchers interested in deploying pre-trained LLMs on mobile devices. Existing efforts in on-device LLM inference have primarily focused on model compression and efficient inference techniques, with a strong emphasis
Coode is available https://githee wtt.://o///J///ee http://mmyyyy.ettttookrraaaaeeee

---

Published as a conference paper at ICLR 2025
deploying models edge SoC (system-on-a- with GPU and Linux systems (Lin al. 2024; Gerganoy 2023; MLC. 2023a Lu et al. 2024, Xu et al., 2024). These approaches aim to achieve desired quality, latency, and throughput while operating within the constraints of the target platform, e.g., available memory and power limitations. However, the challenges and opportunities of efficiently deploying these large models on mobile platforms, such as smartphones (e.g., NVIDIA Jetson Orin Nano), remain largely unexplored.
computers
Current LLM benchmarks primarily target the accuracy of large models on cloud clusters rather than mobile devices (Zheng et al.
2023). Some papers on mobile devices typically examines limited range of models and platforms, often overlooking performance degradation and hallucination due to
quantization (Cöplü et al. 2023 Laskaridis et al. 2024 Murthy et al. 2024).
To bridge this gap, we propose comprehensive benchmarking framework to evaluate the overall user experience of LLMs on mobile devices. This framework automates the evaluation of various LLMs with different compiler options (e.g.. weight and activation quantization) across mobile platforms with diverse hardware capabilities (see Table D. We systematically evaluate each LLM using range of metrics across efficiency, accuracy (generative quality relative to non-quantized models), and harmful output dimensions. Our primary focus is on inference efficiency on mobile platforms, evaluating the computation usage (CPU and GPU), latency, throughput, energy consumption, and memory footprint of different models different platforms (as shown in Figure D Along the accuracy dimension, we evaluate quantized models on mobile devices using mainstream datasets like Natural Questions (Kwiatkowski et al.. 2019) and SQuAD (Rajpurkar al. 2016), employing exact match and perfect match metrics to quantify performance degradation and ensure proper functionality for basic use cases. Also, we evaluate the harmful outputs including hallucination and toxicity of LLMs with existing benchmark datasets Li et al. (2023b); Lin et al. (2022); Luong et al. (2024).
Table 1: Summary of LLMs, mobile platforms, and quantization configurations explored by our benchmark.
LLaMa-2
Touvron el
2023b
LLaMa-3/3.1
(ubeey et al.
2024.
RedPujuma
Computer 2023
LLaMa-3.2 Llama
2024).
Vicuna Chiang et al.
2023
TinyLlama (hang et al.
2024
LLMs
Qwen2 Bai et al.
2023
Mistral-7B (Jang et al.
2023a
Gemma Team
2024
Deepseck-RI -wo--------------di-iiill-
DeepSeck-A1
2025
Phi2
Javaheripi et al.
2023
Phi3 Abdin et al. 2024
Google Pixel
Pixel Sa Pixel
iPhone 15 Pro, iPhone 12 Pro, S22 Ultra
Mobile Platforms
Orange Pi
Nvidia Jetson Nano Nano
2-bit (MI.C, llama.cpp).
Quantization
4-bit (MLC, llmma.cpp). S-bit (lama.cpp), 6-bit (llama.cpp). 8-bit (MLC, Ilama.cpp)
Frantar et al. 2022 Lin et al., 2024 Li et al. 2020: Dettmers & Zettlemoyer 2023
Unlike existing benchmarking efforts for LLM deployment on mobile devices, our study explores and evaluates feasible pre-trained models with the most popular quantizations. We highlight various combinations of L.LM configurations suitable for mobile deployment (Zhang et al. 2024 Computer
2023 Team 2024 Llama 2024; Abdin al. 2024). along with several available quantization options (in et al. 2024 Frantar et al. 2022 2023a Dettmers& & Zettlemoyer 2023).
We found that 4-bit quantization methods, such as group-wise YYang et al. 2024). GPTQ Frantar et al. 2022), and AWQ (Lin et al., 2024), can generally preserve the performance of LLMs while reducing their size one-quarter the original non-quantized model. This configuration diversity can potentially lead developers (and users) to make sub-optimal choices in terms of performance and efficiency. Our benchmarking analysis, focused on resource usage during inference, provides insights into efficient deployment strategies tailored for mobile platforms. These strategies include joint considerations of 1) model architecture, 2) quantization strategy, and 3) model size. In summary, the major contributions of our paper are:
To enable a comprehensive evaluation of various LLMs, we first develop lightweight, auto- mated benchmarking framework that collects performance metries from mobile devices via USB, eliminating the need for additional equipment.
We evaluate various quantized LLMs on mobile platforms with different hardware capabilities, measuring their resource utilization, power consumption, throughput, and inference latency.
2

---

Published as a conference paper at ICLR 2025
Perfermac Narr
Thougheut
SQuAn
Host
Fonct Sfoco
Exeecylin
Datasets
InstfaaAA
Natumal Quorizes
Astonutton
Cintol
MTltera
Exaleater
Pronue
Profiling and Moriter
Cenera-
Despeddoli
Quarliation
Gorprato
GPTQ
Thay
Viare
iPtaa
Schedsler
GOMIL
AWG
Qum2
Viusi -ii-eome
Model
Blgs
Kiesacli Hiid
Fatter
Relbyom
Quandioe
Sai pai Adai 4-b
Framework
MLC
lkana.cpp
OS
Andovii
105
Ulam
System Bassaroo
INIDIA
RAM
Platform
Andhoid
iPhone
Ouange
Jesan Orin
Figure I: Overview and workflow of PalmBench our evaluation and benchmarking framework for Large Language Models (LLMs) mobile devices.
We validate the knowledge and answering accuracy of quantized models compared to their non- quantized counterparts, and investigate the potential issues of compressed models such as toxicity, bias, and the generation of erroneous or random outputs (hallucinations).
Finally, our benchmarking leads several key observations, highlighting the quantization dif- ferences across models, platforms, and frameworks. We also observed that the iOS platform outperforms others power efficiency, latency, and throughput for LLM inference
RELATED WORK
Our benchmarking study extensively evaluated prior efforts focused on optimizing LLMs for mobile devices. These efforts include frameworks (MLC) 2023a b) Gerganov, 2023), the development of
smaller models (Computer) 2023 Abdin et al. 2024
al. 2023c), and model quantization
techniques (Frantar et al.) 2022).
Large Language Models. LLMs like ChatGPT (OpenAl 2023) the Llama series (Tovroo et al. 2023a o Dubey et al. 2024; Llama 2024), Mistral (Jing et al.) (202a), Vicuna (Chiang et al. 2023).
Gemma (Team) 2024
are gaining substantial influence generative Al applications. Their in terms of power consumption and memory
linearly with model size, introducing significant operational overhead during inference. This challenge has catalyzed research into efficient on-device LLM deployment (Xu et al. (2024] Lin et al. 20224] Zhu al. 2023 Lu et 2024), with particular emphasis on model compression and optimization. Recent advances in compact model architectures have yielded promising alternatives, including Google's Gemma-2-2B (Team] 2024). Llama-3.2- BB3B (Llama 2024), RedPajama-INCe -NCTTE-- (Computer) 2023). Phi-2/Phi-3 (Abdi et al. 2024), and Tiny Llama (Zhang et al. 2024).
resource requirements-
Quantization. Post-training quantization (PTQ) compresses LLMS after full training, creating smaller models optimized for inference. This method enables more efficient storage and faster computation. Group-wise quantization (Yang et al. 2024) partitions neural network weights into groups, quantizing each independently. This approach better aligns with weight distributions, reducing I/O costs and improving performance on mobile platforms. GPTQ (Frantar et al. 2022) further enhances efficiency by compressing LLM weights to 3 or 4 bits compared the standard 8- quantization. Activation- aware Weight Quantization (AWQ) (Lin et al. (2024) identifies and preserves a subset of model weights with larger activation magnitudes, known salient weights, minimize quantization loss.
We have released the code of framework facilitate reproducibilityy and extensions of research.
3

---

Published as a conference paper at ICLR 2025
Prior research (Dettmers & Zettlemoyer 2023 Huang et al. 2024b Lu et al. 2024 Xu et al. 2024
has highlighted significant performance differences among quantization algorithms, showing that RTN lags behind GPTQ and AWQ.
Inference Engine. While numerous efficient inference frameworks exist, MLCC 2023a b) stands out by enabling users develop, quantize, and deploy LLMs across diverse platforms, including mobile devices and web browsers. Llama.cpp (Gerganov) 2023), written in C++, offers lightweight, portable alternative Python-based frameworks. It supports multiple GPU kernels for high-speed processing and focuses on mixed quantization methods, particularly K K-Quants. Both frameworks provide pre-compiled models while allowing users to perform custom quantization as needed.
Table 2: Metrics for evaluating the performance of LLMs on mobile devices. Memory usage includes both the model loaded to the memory and the framework program running devices.
Metric
Definition
CPU Utilization (%)
Percentage of the total processor cycles consumed by LLM
GPU Utilization (%)
Percentage of the total GPU computing resource during LLM inference
Memory Footprint (GB)
Measurement of main memory used by the LLM application
Memory Utilization (%)
Percentage of main memory used by the LLM application Number of oulput tokens per second generated by LLM
Throughput (Tok (s)
Output Matching
Accuracy degradation of the compressed model relative to the original model
Toxicity
Toxicity score calculated on 25k sentences by Perspective API
Hallucination (%)
Percentage of erroneous or random outputs not related to the questions
Benchmark.
Most existing benchmark frameworks of LLMs focus on maximizing performance across different model architectures and evaluating a model's general world knowledge, question-
answering, and reasoning ability (Zheng et al. 2023 Hendrycks et al. 2021 Rajpurkar et al. 2016)]
Kwiatkowski et al., 2019). Some existing studies evaluating LLMs for mobile deployment limited in scope (Murthy et al.] [0244] Coplü et al. 2023; Laskaridis et al. (024) or evaluating online
models instead of on-device inference (Lee et al.) 2024). They fail to comprchensively examine
resource efficiency and power consumption, which are crucial for mobile deployment. Recent efforts to benchmark edge L.LMs include the Starting-kit competition framework (Huang et al. 20024a), built
on MLC (MLC. 2023a b). which facilitates LLM evaluation on Linux-based edge devices.
MELT (Laskaridis al. 2024) evaluates five LLMs across devices, measuring throughput, power, and Q&& accuracy, but lacks comprehensive analysis of resource utilization, energy efficiency across quantization methods, and layer-wise GPU profiling. Similarly. MobileAIBench (Murthy al. 2024) offers comprehensive benchmarks for quantized LLMs and LMMs, with a focus on NLP tasks and resource utilization, but its scope is limited to iOS devices and lacks both automated testing capabilities and cross-plaffom evaluation support. Furthermore, all these existing benchmark works ignore the toxicity, bias, and generation of erroneous or randomized outputs (which always occur in quantized
that significantly impact user experience across different frameworks.
METHODOLOGY
To evaluate the LLMs mobile devices, we created the PalmBench framework, which focuses the following three aspects:
1) Benchmark Automation. We designed an automated framework comprising an Evaluator, Scheduler and Profiling and Monitor modules, designed to handle experimental prompt datasets, evaluate quantized models, and schedule profiling across various metrics. The framework's workflow is illustrated in Fig. [
various platforms-such Our primary focus is on the resource demands of different models across CPU, GPU, and memory that significantly impact user experience. Our study goes beyond resource demands, aiming to quantify the impact of different quantization techniques on both performance and resource efficiency across various satte-of-the-eee models.
2) Resource Utilization.
3) Model Accuracy. Although a model architecture largely dictates its outputs, we observe device- specific variations under different quantization methods. To validate quantized L.LMs and evaluate the
4

---

Published as a conference paper at ICLR 2025
cepppeession-nndueed accuracy drop, we use Exact Match and FI scores, comparing their knowledge and answering accuracy to the original models (calculation methods are detailed in the Appendix(A.3) Additionally, we test these models on standard tasks with open-source datasets such SQuAD and Natural Questions (Rajpurkar et al. 2016 Kwiatkowski et al. 2019). Moreover, we also investigate the potential issues of toxicity and the generation of erroneous randomized outputs that always occur in compressed models and have not been thoroughly studied in previous work (Laskaridis et al.
2024
3.1 METRICS AND DATASETS
Table 2 summarizes all the metrics used in our benchmark. To evaluate the accuracy and correctness of quantized M.Mss, we use popular Question-Asswrring datasets such as SQuAD (Rajpurkar et al., 2016) and Natural Questions (NQ) (Kwiatkowski et al., (2019), comparing their performance with that of the original, non-quantized models. Additionally, we employ comprehensive benchmarks for different tasks, including MTBench (Zheng al. (023) to compare their language understanding
and reasoning capabilities across different quantizations. Also, we measured toxicity by calculating toxic score by using Perspective API (Perspective) 2020) and TET (Luong al.) 2024) and evaluated hallucination in each quantized LLM using HaluEval (Li et al. (2023a) and TruthfulQA (Lin et al.) 2022) benchmarks. The Appendix [A.7 and Appendix [A.9 provide some examples of these datasets for benchmarking. These widely-recognized datasets ensure that our experiments and metrics are both convincing and reproducible.
3.2 CHOICE OF L.MS
We have identified and converted several popular models for benchmarking edge and mobile devices using model weights from their official Huggingface or GitHub repositories. These models converted into experimental formats such GGUF and K-quant for Llama.cpp (Gerganov) (023)
or compiled using TVM (Chen et al., 2018) for MLC frameworks (MLC) 2023a b). Given that mobile devices typically do not exceed 8GB of memory, it is impractical to test models that
too large, as they would surpass these devices' memory capacity. In our benchmark, we evaluated the various LLMs, including Llama-2-7b-chat (Touvron et al.) 2023b), Llama-3-8B-Instrucc (Tou- vron et al. (023c) Microsoft Phi2 (Abdin el al. 2024). Mistral-7B- 7B-nstuet (Jiang et al. 2023a)) RedPajaama INCITE SEUTTE-Cha-225 Chat-7B (Cmputerr 2023). Vicuna (Chiang al. 2023)), Tiny TinyLlama-II1E Chat-v1.0 (Zhang et al.) (024)), and Qwen2 (Bai et al. (023). The prebuilt weights for these models are readily available in the MLC repository, which also offers options for compilation in various configurations.
While the frameworks offer some pre-compiled models through Huggingface or official repositories, certain models still require quantization and compilation for our experiments by ourselves.
3.2.1 MOBILE DEVICES
We evaluate the LLMs on a range of devices with varying hardware capabilities, as listed in Table O in Appendix, including Google Pixel 4 (P4), Pixel 5a (P5), Pixel 7 (P7), iPhone 12 Pro (IP12), iPhone 15 Pro (IP15), S22 Ultra (S22U), Orange Pi 5 (OP5) (Pi), and Nvidia Jetson Orin Nano (Nano) (Nano). covering mainstream operating systems.
3.2.2 INFERENCE ENGINE
We use two frameworks, MLC-LLM (MLC) 2023a b) and llama.cpp (Gerganov) 2023), as inference engines to execute LLMs on devices. Although many frameworks claim compatibility with mobile devices, they often lack support for popular platforms models. MLC-LLM (MLC) 2023a b) and llama.cpp (Gerganov] 2023) are two of the most popular frameworks that support a wide range of platforms and models. Unfortunately, llama.cpp (Gerganov) 2023) is still incompatible with iPhone. Both frameworks utilize default nucleus sampling and identical LLM decoding hyperparameters to ensure consistency: temperature is set 0.2, Top S = 0.9.
3.2.3 QUANTIZATION
The built-in quantization programs of frameworks primarily quantized the different models. Both of MLC and llama.cpp provides tools for users to quantize the models. MLC supports various quantization levels, including non-quantized float-16 16 (q0f16) and float-32 (q0f32). 3-bit quantization (q3f16_1), 4-bit quantization (q4f16 I). and 4-bit AWQ (q4f16 (_wgq). The format qAfB(id) denotes ''' as the number of bits for weight storage and 'B' as the number of bits for activation storage. llama.cpp supports quantization using its GGUF format, which employs type of group-wise quantization known as K-quant and supports more quantization methods (1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit).
5

---

Published as a conference paper at ICLR 2025
3.2.4 PROMPT INPUT
The prompt input is managed by the Evaluator, which extracts tested prompt texts from datasets based on the evaluation tasks. For Linux edge devices, prompts are transferred from Scheduler via USB serial ports, and benchmark scripts are executed through the Command Line Interface (CLI). For iPhones and Android phones, custom-developed apps-built on MI.C's examples (MI.C) 2023a automatically fetch prompts and simulate touch events to interact with the applications.
Prompt texts are from datasets listed in 3.1
3.2.5 BENCHMARK AUTOMATION
Our benchmarking framework automates workflows and collects profiling data via multiple interfaces, coordinating programs on both PC hosts and mobile/edge devices. The Evaluator extracts prompts from datasets and evaluates outputs using mainstream benchmark datasets as illustrated in Figure [] While frameworks provide limited pre-compiled models, quantizing own models is often required. The Scheduler evaluates quantized models with prompt input from Evaluator; and monitors resource usage through profiling tools like Xcode Instruments for iOS and Android GPU Inspector (AGI) for Android. AGI tracks metries such CPU/GPU utilization, memory, energy. and latency, with data transferred via ADB. For iOS, a custom GPU plugin built on IOKit Tan 2018) complements Xcode's profiling capabilities. On edge devices, btop is used for Orange Pi, and jetson-stats for Nvidia Jetson Nano. Both are lightweight and non-intrusive. Apple and Google profiling tools access battery data from the PMU, ensuring precise energy measurements without disrupting device operation. These official tools from Apple and Google are built-in utilities for system monitoring that provide accurate battery usage data directly from the Power Management Unit (PMU). On Ubuntu-hased edge devices, Orange Pi uses btop (aristocratos) (202)) for CPU and GPU utilization stats, while Nvidia Jetson Nano employs the built-in jetson-stats tool. Both methods are non-intrusive and do not disrupt device communication.
3.2.6 GPU DRIVER
Although MLC-LLM (MLC 2023a b) and llama.cpp (Gerganov 2023) support various drivers; OpenCL. is the preferred and most mature GPU driver commonly used for both Android phones and Ubuntu-based edge computing devices. The iPhone utilizes Apple's proprietary Metal driver, which is well supported by both MLC (MLC) 2023a) and TVM (Chen et al. 2018). Nvidia Jetson Nano leverages its own CUDA with a highly optimized driver (Nano).
3.2.7 EQUIPMENTS
In addition to resource usage, also look into energy efficiency, critical factor impacting user experience. We employ two devices to comprehensively evaluate the effects of quantization on power consumption and the distribution of device temperature. For thermal behavior analysis, we utilize the FLIR C5 thermal imaging camera Flir (2020). We also employed professional -SBebased power consumption instruments from Klein Tools to measure the power consumed by each tested device. This enables us to investigate the energy efficiency and thermal behavior of mobile platforms across different models and quantization methods, which are crucial factors affecting user experience.
4
EXPERIMENTS
We present the most significant benchmarking results for LLMs across various models and platforms (outlined in Section here, and provide additional results in the Appendix.
4.1
EXPERIMENTAL SETUP
We evaluate LLMs across various devices (detailed in Section 3 using MLC (MLC 2023a), which
supports Apple, Android, and Linux Edge platforms, and llama.cpp Gerganoy 2023). w which supports Android and Ubuntu systems. Given our focus on mobile deployment, we select models based on practical size constraints. All experiments were repeated 10 times under identical conditions for consistency, using llama.cpp and MLC frameworks with standardized settings: temperature 0.2, Top-S= = 0.9, and 4096-token context window. Devices were factory reset before testing, and evaluations were conducted on the latest OS versions (Ubuntu 14.04.06 LTS, Android 15, and iOS 17.6.1, as illustrated in table in the Appendix).
4.2 RESOURCE UTILIZATION
We first evaluate the impact of quantization on resource efficiency using the MLC and llama.cpp frameworks on Android phones and edge devices for the models detailed in previous sections.

---

Published as a conference paper at ICLR 2025
Laral AM
Waree R1
Tellaw LB
4 0.k
M.K.M
4 M.5-
(a) MLC
(b) llama.cpp
Figure Average memory usage (GB) while running MLC and laam..pp.
CPU Unitzaiion (56)
GPU Unlizztioon (%)
MLC Ail6-AAQ AAIG-MLC C 4416-GPTQ
80%
80%
60%
60%
40%
40%
20%
20%
P4
P5a
P7
S22U
IP12
IP15
OP5
Namo
P4
P5a
P7
S22U
IP12
IP15
OP5
Nano
(a)
CPU
(b) GPU
Figure 3: CPU and GPU usage during inference RedPajama--NNITTT-B across different quantiza-
tions.
Memory Utilization: LLM inference is inherently memory-bound, and its memory utilization can be reduced significantly by quantization, which reduces the precision of weights and activations.
To ensure the models fit within the tested devices, including iPhones (12 Pro, 15 Pro) and Android phones (Pixel 4/5a/7, Samsung S22 Ultra), we evaluated only 3-bit and 4-bit quantized models in the MLC framework for total memory usage. Higher-bit models exceeded the devices' memory capacity. Figures 2[a) and [(b]show show average memory consumption across platforms for both MLC and llama.cpp frameworks. While total memory usage remains consistent for each combination, we observe platform-specine
variations. Memory usage scales with quantization levels, with sub-4-bit quantization offering significant reductions. MLC shows lower memory footprint iPhone compared to Android, while Jetson Orin Nano's CUDA implementation outperforms Orange Pi's OpenCL (Figure 6). llama.epp generally consumes less memory than MLC across platforms, likely due to its lightweight C++ implementation and CLI interface.
CPU and GPU Utilization: LLM execution depends the computational resources of mobile platforms, with CPUs handling data transfer and model offloading between memory and GPUs, which are primarily used for inference. Both MLC and Ilama.cpp supports GPU-based model inference. We measured CPU and GPU activity to evaluate how quantization impacts memory traffic and GPU workload, as shown in Figure 3 The results indicate that CPU and GPU utilization varies across models and platforms. Notably, 3-bit quantization reduces resource usage, likely due to decreased memory data transfers and a lighter GPU inference workload. Furthermore, iPhones demonstrate lower resource utilization than other platforms, showcasing Apple's exceptional efforts in hondware-softwan optimization and compatibility. We also gathered GPU traces through automated benchmarking tools and charted the GPU utilization timeline to examine how GPU workloads vary when running identical models with diffcrent quantization methods, as depicted in Figure 4 The findings reveal that models with b-tit quantization utilize more GPU duty cycles than those with 3-bit quantization, thereby consuming more GPU time. Additionally, higher quantization requires increased energy and computational resources for inference. To explore how quantization affects GPU memory read and write operations, Figure 5 fillustrates the memory throughput for read and write operations the GPU while operating B-a--aa---B----tttre
and LLaMa-3-8BB ----Instruct-4fft
RodPajama-3B--a 3B-g4f1
RodPajama-3B
100%
GPU Usage (%))
60%
40%
20%
Time ims x 10
Figure 4: GPU Utilization (%) timeline for 3-bit and 4-bit quantized RedPajama models on Google Pixel 7.
7

---

Published as a conference paper at ICLR 2025
GPU write
GPU read
GPU write
GPU read
15
Speed (GB(s)
Speed SpoeddGBis) (GBis)
10-
10
5
4
Time (ms 10")
Time imis 10+)
(a) Llama-3-8B-33f16
(b)
Figure GPU memory read/write speed while running LLaMa-3-8B. Instruct in 3-bit and 4-bit
quantization on 7.
Memory Usage (GB)
8
Memory Usage (GB)
G3116-MLC
MLC
g3116-MLC
g MLC
6
4
iPhone
Android
OrangePi
Nano
iPhone
Android
OrangePi
Nana
Llama-3-S8 SB for all platforms
(b) Gemma 2B for all platforms
Figure 6: Measured memory usage (GB) across different platforms using Llama-3-8B and Gemma- 2-2B by MLC LLM to compare the memory usage between large model (Llama-3-8B) and small model (Gemma-2-2BB
The operation of LLaMa-3-40 3-88--8BBBBB- ABB-Intrruct-q4f16 demands additional GPU workload and writing cycles. This observation confirms the hypothesis that higher quantization increases GPU memory data usage, with inference performance constrained by memory throughput. We analyzed GPU resource usage through memory breakdown and layer-by-layer profiling across two transformer blocks to identify potential bottlenecks in LLM inference on mobile devices. Results are shown in Appendix Fig. TT Higher-bit quantizations often achieve higher GPU utilization due their increased precision. Nonetheless, low-bit models still fully utilize GPU resources through operations like matrix multiplication and memory access. Therefore, optimizing matrix multiplications and transformer is critical to reducing GPU usage.
Thaude (Taa)
Prita
Palli
80-
thragton
Deadds
Daadd
Poods
80.
1lama2-7B-qu3
(b) Llama3-88 -B-333-
(c) Phi2-q4f16
(d)
16
Figure 7: Prefilling and decoding throughput (tok/s) for Llama2-7B-gee 7B-64 Llama3-8BB
8B-13f16 and
RedPajama-3B- -B-4466...
4.3 PREFILLING AND DECODING THROUGHPUT
We analyzed prefilling and decoding throughput (tok/s), key factors influencing user experience. Higher throughput and lower latency reflect faster model outputs. Figure 7 presents the pre-filling and decoding throughput for Llama2-7B-q4f16 Phi2-q4f16, Llama3-8B-q31 -00---8--331.-6- and RedPajama-3B
q4f16. Appendix Figure 13 shows throughput across platforms using MLC. Results indicate that smaller models, such as RedPajama-INCrie 3B and M.LLLMM--I.1B achieve higher throughput and execute faster on mobile devices. Moreover, the results indicate that iPhones, particularly when running Llama-2-7B and Llama-3-8B models, deliver significantly higher throughput compared to other devices. Even the three-year-old iPhone 12 Pro outperforms newer Android devices and Nvidia's 's Jetson Orin Nano in maintaining relatively high throughput, demonstrating metal-accelerated inference performance. When running Mistral-7B- BB-43116 (Jiang et al. 2023b) and Phi2-q4f16, which similar in size but differ in parameter scale and quantization levels, significant differences in prefilling and decoding throughput are observed (Figure 7 and Figure [C)). Models with fewer parameters and higher-bit quantization, like Phi2-q4f16 6, decode faster and consume less total memory,
8

---

Published as a conference paper at ICLR 2025
including KV cache and overhead (Appendix Figure 12). This efficiency makes smaller models like Phi2 more suitable for mobile devices.
4.4 OUTPUT MATCHING AND CORRECTNESS
Quantization often compromises model accuracy, particularly when using lower-bit representations. To validate the correctness and assess the performance degradation of quantized models, we use question data from the SQuAD (Rajpurkar et al. 2016) and Natural Question (Kwiatkowski et al. 2019) dataset to calculate the Exact Match and Fl score, using the output of the original non-quantized model as a reference. The exact match and Fi score results are shown Figure Moreover, our observation also shows that 4-bit and 6-bit quantization mostly maintains performance compared to original non-quantized model, with 4-bit quantization requiring less memory and computational resources (Figure 2(a) 2(b) ). Interestingly, 3-bit quantization used by MILC achieved no significant advantage over the 2-bit models Ilama.cpp. while the 5-bit model performed similarly to bit models (MLC and llama.cpp) but consumed more resources. We suspect that the particular implementation and optimization the tested version caused the 5-bit and 3-bit models to degrade more than another quantization variant.
Exact Match (%o
100%
Laam3333,1 1-888
BBB
Phi2
RedPajjam--Cher Chal-3BB
Mistral-7B
Vicuna-78 7B
Qwen2-78
80%
40%
20%
88ml 2-bit
3-bil
4-bil
S-bit
6-bit
GPTO 3-bi
GPTO 4-bit
AWO 4-hit
Ft 4-bit
(a) Exact Match
Llamm2-7B
RedPajame Chat-3B
Vicuna-1 7B
Qwen2-7B
Exact (N)
100%
L/3me3333 1 1-8B
Phi2
Mistral-7B
80%
60%
40%
20%
agmal 2-bit
3-bil
4-bit
5-bit
6-hii
GPTQ 3-bir
GPTO 4-bit
AWQ 4-bii
FT 4-bit
(b) Fl score
Figure 8: Scores of exact match and F1 score to examine the performance loss after models are quantized.
11-M62-78-116
S K.
Llama-2-7b--e 2-bb-chat-hh--ggeee
LL LLaMa2-78-4 K.KK
LLaMA2-TB-44 GPTO
LLaMu2-78-qq AWQ
LLaMA3-7B4 K
- LLLMM2-78443.KK K.M
LLLMM2-TB-4 3. K.M LLaMo2-7B QPTQ
Reasening
LLMMI2-11--3. _WQQ LLMMM0-710-gLKK..
1aam--2-bbchutth---fff..
Marh
Rolephay
Reasoning
Marh
Roleplay
Mari
Rolelay
Coding
Wriling
Coding
Wiitin
Coding
2
6
10
Writing
Extraction
Hummanities
Farmation
Harronitos
Extraction
Humanitics
STFM
(a) K-quant used by llama.cpp
(b) K-quant vs. AWQ GPTQ (3- bit, 4-bit)
(c) MLC
Figure 9: MTBench scores in different categories using Llama2-7B with various quantizations.
4.5 TASKS
To evaluate the performance of quantized models across various tasks, we utilize MT-bench (Zheng et al. 2023). which employs predefined multi-turn question set to evaluate models across eight categories. Figure shows that models with higher bit quantization generally achieve better scores across all categories. In contrast, lower-bit quantization still performs well in humanities, writing, and extraction tasks. For users with devices that have limited resources, particularly those with less than 4GB of memory. 2- 3-bit quantization can still provide an adequate user experience in these tasks or for simple Q& A applications.

---

Published as a conference paper at ICLR 2025
Table Evaluation of temperature and power consumption during inference Llama3-8B across different mobile phones
Llama-3.2-- 3.23-- 3-bit quantization
Platforss
Pixel
Pixel Sa
Pixel
S22 Ultra
iPhone 12 Pro
iPhone 15 Pro
Orange Pi
Jetson Nano
Temp. (*)
Temp. ())
28.3
28.7
28.5
27.2
27.2
25,3
47.5
43.3
Power Consumed (mWh)
13.32
12.98
14.54
13.25
11.21
10.13
25.4
22.3
Llama-3.2-38 4-bit quantization
Platforms
Pixel
Pixel Sa
Pixel
S22 Ultra
iPhone 12 Pro
iPhone 15 Pro
Orange PL
Jetson Nano
Peak Tamp. (*)
53.1
54.8
52.6
48.7
47.2
46.3
75.4
69.5
Arg. Temp.
30.3
45.3
Power Consumed (mWh)
14.23
13.51
14.68
15.26
13.12
13.05
27.8
25.6
Table 4: Evaluation of Hallucination Outputs across Different Quantization Levels in Llama3-8B.
Quantization
2-bit
3-bit
4-bit (GPTQ)
8-bit
4-bit (ggml)
4-bit (AWQ)
4-bit (FT)
Halucination 32.7%
37.5%
9.1%
8.1%
12.5%
8.9%
8.7%
TruthfulQA 76%
73%
92.1%
91.4%
90.1%
92.3%
91.5%
Toxicity
46.243
64,098
28.679
23.965
41.107
30.072
29.405
Table 5: Evaluation of Hallucination Outputs across Different Quantization Levels in Gemma-2-2B.
Quantization
3-bit
4-bit (GPTQ)
4-bit (ggml)
(AWQ)
4-bit (FT)
Halucination
34.2%
32.5%
9.1%
7.9%
14.5%
8.9%
8.7%
TruthfulQA 72%
73.2%
91.1%
92.4%
84.5%
89.3%
90.5%
Toxicity 36.121
65.22
25.045
23.102
24.215
31,202
25.455
4.6 POWER CONSUMPTION AND TEMPERATURE
Model quantization dramatically reduces memory usage and GPU execution time, LLM infer- ence is largely memory-bout One interesting observation is that quantization also impacts power consumption and device temperature on mobile platforms, as shown in Table With 4-bit quantiza- tion, higher resource usage leads increased temperature and power consumption, with the 4-bit Llama-3.2-3B model consuming 25.2% more power than its 3-bit counterpart.
4.7 HALLUCINATION AND TOXICITY
LLMS can potentially produce incorrect or harmful information, particularly hallucinated and toxic content. We evaluate Toxicity and Hallucination using GPT-4o OpenAl (2023) and Claude-3.5- Sonnet Anthropic (2023) through an LLM-as-a-judke approach, as shown in Table 4 and Table Lower-bir bit quantization typically increases hallucinations and toxicity. Notably, 3-bit quantization occasionally performs worse than 2-bit group- wise quantization and all 4-bit methods, resulting in more hallucinations and toxic outputs. Initially, we believed the model and quantization algorithms primarily caused hallucinations. However, recently discovered that inference framework imple- mentations and mismatched parameters can also lead to hallucinated or random outputs. Examples of hallucinated and toxic outputs are provided in Tables 11 in Appendix A.9
5 CONCLUSIONS
We present a comprehensive benchmark for evaluating LLMs under various quantization schemes on diverse mobile platforms. Our lightweight, all-in-one automated benchmarking framework enables users to evaluate mobile devices via USB, providing extensive metrics and datasets. This study uniquely focuses on resource efficiency for mobile GPUs, contrasting with traditional high-end GPU cluster evaluations. Key findings highlight the superiority of iOS platform in energy efficiency and throughput, and quantization' 's effectiveness reducing resource requirements. We also examine accuracy and potential issues in quantized models, including toxicity and erroncous outputs. This research provides crucial insights for efficient LLM deployment in mobile environments, addressing previously overlooked aspects of on-device L.LM performance.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 Technical Re- port: A Highly Capable Language Model Locally on Your Phone. arXiv preprint 1eX.i:22444.12119 2024.
Anthropic.
Introducing the next generation of Claude. httD::/www.antthroiic.cccmmm// claude- 3- 3-family. 2023.
aristocratos. btop: A Monitor of Resources. http:///ggtuub.ccomrrrrsttrrrrre
2021.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Gc, and otehrs. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
Tianqi Chen, Thierry Morcau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Meghan Cowan, Haichen Shen, Leyuan Wang. Yuwei Hu, Luis Ceze, Carlos Guestrin, and Arvind Krishnamurthy. TVM: An Automated End-to-End Optimizing Compiler for Deep Learning. In In Proc. of the 13th OSDI, 2018.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric Xing. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90% ChatGPT Quality, March 2023. URL
http::://mmsss.org////o/22233333 00t-:2000200333--ccuuaaa
Together Computer. RedPajama: An Open Source Recipe to Reproduce LLaMA training dataset, computer/Reddaajaanee 2023. /ttp::tss///ittuuuu com/ htwp://tterrcmmmteeee/eeeeeeeeee
Data
DeepSeek-AI. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, 2025.
Tim Dettmers and Luke Zettlemoyer. The case for 4-bit precision: k-bit Inference Scaling Laws, 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad AI-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, al. The llama herd of models. 2024. URL httpss//arxii http://axxv..o.r/aaaaaa abrg//bb//a0/ 2407. 21783
Flir. FLIR C5 Compact Thermal Imaging Camera. https: http://wwww.....mo/rrruuttttete. ?vertical=condi wtw.:oooniiiiiioooiiiiiioooooooi toringgsegmentt
2020.
Elias Frantar, Salch Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: Accurate Post-training Compression for Generative Pretrained Transformers. arXiv preprint 1rXiv:2210.17322. 2022.
Georgi Gerganov. Ilama.cpp. httpss//ggthioo.ccm//ggrr//////a 2023,
Github. Copilot. http:://ggthuu.ccc//eettturrss/p
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring Massive Multitask Language Understanding, 2021.
Qinghao Hu, Zhisheng Ye, Zerui Wang, Guoteng Wang. Meng Zhang, Qiaoling Chen, Peng Sun, Dahua Lin, Xiaolin Wang, Yingwei Luo, Yonggang Wen, and Tianwei Zhang. Characterization of Large Language Model Development in the Datacenter. In 2/st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24). USENIX Association, 2024.
Tienjin Huang, Songyang Zhang, and Shiwei Liu. Competition, 2024a. EdgeleviicLLLMCmmetttioioo www.eettiiitttttttnnnntn- Starting-Kit
Starting Kit for Edge-
Device LLM
URL httpps///iii....m/mi/iiiinnnnn wow.ubb.coommmaaninnlllooo
Wei Huang, Xingyu Zheng, Xudong Ma. Haotong Qin, Chengtao Lv, Hong Chen, Jie Luo, Xiaojuan Qi. Xianglong Liu, and Michele Magno. An Empirical Study of LLaMA3 Quantization: From
L.MMs to ML..Ms, 2024b.
11

---

Published as a conference paper at ICLR 2025
Mojan Javaheripi, Sébastien Bubeck, and Marah Abdin. Phi-2: The surprising power of small language models. In Proc. of 37th NeurlPS, 2023.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7B. arXiv preprint arXiv:2310.0682. 2023a.
Albert Qiaochu Jiang, Alexandre Sablayrolles, Arthur Mensch, al. Mistral 7B. ArXiv, 2023b.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, et al. Natural Questions: a Benchmarrk for Question Answering Research. Transactions of the Association of Computational Linguistics, 2019.
Stefanos Laskaridis, Kleomenis Katevas, Lorenzo Minto, and Hamed Haddadi. MELTing point: Mobile Evaluation of Language Transformers. CoRR, abs/2403, 12844, 2024, doi: 10.48550/ ARXIV 12844. URL
2403. 12844
Juyong Lee, Taywon Min, Minyong An, Changyeon Kim, and Kimin Lee. Benchmarking Mobile Device Control Agents across Diverse Configurations. In ICLR 2024 Workshop on Generative Models for Decision Making, 2024.
Junyi Li, Xiaoxue Cheng, Xin Zhao, Jian- Jian-Yun Nie, and Ji-Rong Wen. HaluEval: Large-Scale Hallucination Evaluation Benchmark for Large Language Models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP, pp. 6449-6464. Association for Computational Linguistics, 2023a. doi: 10. 1/233//1/222..MMLPPPP URL httptsh///o......./ org/10. 18653/v1/2023 http://2222..m.mmmmammmmmmm mmnlp-main... 397
Junyi Li, Xiaoxue Cheng, Xin Zhao, Jian-Yur Nie, and Ji-Rong Wen. HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models. In The 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP 2023), 2023b.
Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need II: phi-1.5 technical report. CoRR, abs/2309 2023c. doi: 48550/ Xiv. 2309. 05463
10.48550/ARXIV.2300055544
URL WRLhhttps::2 http::::/ddoi..rrg....
Yuhang Li, Xin Dong. Sai Qian Zhang, Haoli Bai, Yuanpeng Chen, and Wei Wang. Rtn: Reparame- terized ternary network. In Proceedings of the AAAI Conference Artificial Intelligence (AAAI 2020), volume 34, pp. 4780-4787, 2020.
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration. In MLSys, 2024.
Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Mcasuring How Models Mimic Human Falsehoods. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2022.
Meta
kttps://alt memta.combblll 1lama- 3- 2onnocoonnttt22222 www.mmett22222iiiii---ee-------- moleeeeeiiiieeeevieeesss// 2024.
Llama.
Introducing Llama 3.2.
Zhenyan Lu, Xiang Li, Dongqi Cai, Rongjie Yi, Fangming Liu, Xiwen Zhang, Nicholas D. Lane, and Mengwei Xu. Small Language Models: Survey, Measurements, and Insights, 2024. URL https://arxiv.
15797 15790
Tinh Luong, Thanh-Thien Lc. Linh Ngo. and Thien Nguyen. Realistic Evaluation of Toxicity in Large Language Models. In Lun-We Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics ACL 2024, pp. 1038 1038-1047. Association for Computational Linguistiis,, August 2024. doi: 10. 10..8663////22224innnnene......
MLC. Machine Learning Compilation
2023a.
MLC. MLC-LLM Github httpps//ggittuu..ommmmmcaaaaaaa
2023b
12

---

Published as a conference paper at ICLR 2025
Rithesh Murthy, Liangwei Yang. Juntao Tan, Tulika Manoj Awalgaonkar, Yilun Zhou, Shelby Heinecke, Sachin Desai, Jason Wu, Ran Xu, Sarah Tan, Jianguo Zhang, Zhiwei Liu, Shirley Kokane, Zuxin Liu, Ming Zhu, Huan Wang, Caiming Xiong, and Silvio Savarese. MobilcAlBench: Benchmarking LLMS and LMMs for On-Device Use Cases, 2024.
Jetson Orin Nano. jetson-nano
Nvidia Jetson Orin Nano.
OpenAL. GPT-4 technicallalacccc report. CoRR, abs/230000033 2023. doi: 10445500AARXXIV22000007777
Perspective.
Perspective API. http:///gttt...occccceststttttrt 2020.
Orange Pi.
Ccapptteehoolccooooooooeeeeeaaas http://www.oaraggi..oogghhaaaaaa
Orange Pi 5B.
Pranav Rajpurkar, Jian Zhang. Konstantin Lopyrev, and Percy Liang. SQuAD: 100, 000+ Questions for Machine Comprehension of Text. In EMNLP, 2016.
Shaden Smith, ostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari,, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, Elton Zhang, Rewon Child, Reza Yazdani Aminabadi, Julie Bernauer, Xia Song. Mohammad Shoeybi, Yuxiong He, Michael Houston, Saurabh Tiwary, and Bryan Catanzaro. Using DeepSpeed and Megatron to Train Megatron- NL. 530B, A Large-Scale Generative Language Model, 2022.
Ricky Tan. GPUUtilization. http:://gtthhboocoo//c/ccaaaaaut
2018.
Gemma Team. Gemma 2: Improving open language models at practical size. ArXiv, abs/2408.00118. 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anne Lachaux, et al. LLaMA: Open and Efficient Foundation Language Models. ArXiv, abs/2302. 13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti, et al. Llama 2: Open Foundation and Fine-Tuned Chat Models, 2023b.
Hugo Touvron, Louis Martin, et Llama 2: Open Foundation and Fine-Tuned Chat Models. ArXiv, 2023c.
Jiajun Xu, Zhiyuan Li, Wei Chen, Qun Wang, Xin Gao, Qi Cai, and Ziyuan Ling. On-Device Language Models: A Comprehensive Review, 2024. URL http::/:://rrr http://aaxii...orga////////// 02088
Jiaming Yang, Chenwei Tang, Caiyang Yu, and Jiancheng Lv. GWQ: Group- Wise Quantization Framework for Neural Networks. In Asian Conference Machine Learning. PMLR, 2024.
Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. TinyLlama: An Open-Source Small Language Model, 2024.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li. Dacheng Li, Eric. Xing, Hao Zhang. Joseph E. Gonzalez, and Ion Stoica. Judging L.LM-as-a-judge with MT-Bench and Chathot Arena, 2023.
Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang. A Survey on Model Compression for Large Language Models, 2023.
Tolga Çöplü, Mare Loedi, Arto Bendiken, Mykhailo Makohin, Joshua J. Bouw, and Stephen Cobb. A Performance Evaluation of a Quantized Large Language Model on Various Smartphones, 2023.
13

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 CONFIGURATION AND DECODING STRATEGY OF FRAMEWORK
Although llama.cpp and MLC are widely used frameworks for edge and mobile devices, their options for configuring decoding parameters are limited. In our experiments, we ensured consistency by setting identical hyperparameters for both frameworks: temperature randomness in model outputs, and Top K was set to 40. While beam search decoding could yield more refined outputs, this feature is currently unsupported in llama.cpp.
fixed at 0.2 to preserve some
1) Temperature Temperature is hyperparameter controlling the randomness of generated text by adjusting token probability distribution. Higher values (e.g., 1) yield more diverse outputs, while lower values (e.g., 0.1) produce more focused and deterministic responses. We set the default temperature to 0.4, striking a balance between creativity and consistency.
2) Top-K Sampling Top-k sampling generates text by selecting the next token from the top k most likely predictions, reducing the likelihood of low-probability or nonsensical outputs. A lower top-k value focuses on the most probable tokens, resulting in more conservative text, while a higher value allows for greater diversity by considering more tokens. We mainly use Top-S sampling, Top-K an option. The top-k value in our configuration is set to 40 if needed.
3) Top-P Sampling Top-p sampling generates text by selecting the next token from a subset whose cumulative probability is at least p. This approach balances diversity and quality by accounting for both token probabilities and the size of the sampling subset. Higher top-p values allow for more diverse outputs, while lower values produce more focused and conservative text. In our configuration, top-p value is set to 0.9.
A.2 SPECIFICATIONS OF TESTING DEVICES
We evaluate the L.LMs on range of devices as listed in Table including Google Pixel 4 (P4), Pixel 5a (P5), Pixel 7 (P7), iPhone 12 Pro (IP12), iPhone 15 Pro (IP15), S22 Ultra (S22U), Orange Pi 5 (OP5) Pi and Nvidia Jetson Orin Nano (Nano) Nano) covering mainstream operating systems.
Table 6: Mobile and edge devices for evaluation.
Device
SoC
Memory (GB)
Framework Support
iOS 17.6.1
iPhone 12 Pro
A14 Bionic
6GB
MLC
iPhone 15 Pro
A17 Bionic
8GB
MLC
iPhone 16 Pro
A18 Pro
8GB
MLC
Android
Pixel 4
Snapdragon 855
6GB
MLC/llama.cpp
Pixel 5a
Snapdragon 765G
6GB
MI.C/llam..pp
Pixel 7
Exynos 5300
8GB
MI.C//lama.cp
S22 Ultra
Snapdragon 8 Gen 1
8GB
MLC/llama.cpp
Uhuntu 14.04.06 LTS
Orange Pi 5
RK3588
8GB
MI.C/llam..cp
Jetson Orin Nano
NVIDIA Orin
8GB
MLC/lamma.cp
A.3 CALCULATION OF EXACT MATCH AND Fl SCORE
Exact Match is calculated as the percentage of predictions that exactly match the ground truth answers. The formula given by:
EM Number of Exact Matches
Total Number of Examples x 100.
14

---

Published as a conference paper at ICLR 2025
Where:
Exact Match: A prediction is considered an exact match if it exactly matches the ground truth answer after normalization (e-g.. removing punctuation, converting to lowercase, or stripping whitespace, depending on implementation specifics).
Total Number of Examples: The total number of examples in the dataset.
F1 Score is metric used to evaluate a model's performance, particularly in classification tasks. It is
defined as the harmonic mean of precision and recall, thereby balancing their trade-off:
Precision X Recall
F1 2 x
Precision + Recall
Where:
True Positives (TP)
Precision
True Positives (TP) + False Positives (FP)
True Positives (TP)
Recall
True Positives (TP) + False Negatives (FN)
A.4 MOBILE DEVICE TEMPERATURE
The mobile phone temperature distribution is measured by FLIR Flir (2020). While power con- sumption provides strong, deterministic relationship with temperature, FLIR imaging offers unique insights into thermal heterogeneity across the device surface. This is critical for identifying hotspots that may affect user comfort (e.g., in contact areas like the back of a phone) or hardware reliability.
Peak Temp. 52.8"
Figure 10: Temperature hile Google Pixel is running Llama2-78 BB-2-strctt (3-bit).
A.5 LAYER-WISE GPU RESOURCE USAGE ANALYSIS AND GENERAL THROUGHIPU (TOK/S) ACROSS ALL MODELS
We conducted a detailed, layer-by-layer analysis of GPU utilization across two Transformer blocks. Our findings show that both self-attention and ferd-forward network (FFN) layers consume the majority of GPU resources. Notably, self-attention becomes increasingly demanding with longer input sequences, driving GPU utilization beyond 92% for self-attention and 95% for FFN- potentially slowing overall inference.
In terms of time complexity, self-attention scales O(n2d), while the FFN layer scales O de), where n is the sequence length and d is the hidden dimension. Consequently, optimizing both the underlying matrix multiplications and the overall Transformer architecture crucial for reducing GPU usage. To optimize GPU resource usage on mobile devices, we can consider some options:
Approximate Attention: Replacing standard self-attention (O(n2) with sparse, low-rank, kernel-based approximations helps alleviate quadratic complexity for long sequence inputs.
Especially in auto-regressive decoding, KV caching and malti-token butching can reduce repeated computation within self-attention layers.
Batching and Caching Optimizations:
Hardware-free -rrieddy-friendly Approaches: Tailor operations (e.g., fused kernels for the FFN block, efficient memory layouts) to specific GPU or mobile-accelerator architectures, thus reducing overhead and latency.
15

---

Published as a conference paper at ICLR 2025
Actralen
Menory Usage (GBB)
OPT Lilosii EVD
4708 (1.95)
652
340
Quantization
(a) Memory Breakdown
(b) Layer- wise GPU Utilization
Figure 11: Analysis of GPU resource utilization for Llama-3-8B on Google Pixel 7: Memory consumption breakdown and GPU utilization across layers.
Activations Model Weightt
Memory Usage (GB)
12
PAIN
Mistral-7--q2166
Phil-q4016
Figure 12: Memory usage comparison: Phi2 4-bit vs. Mistral-7B 3-bit.
A.6 CPU, GPU, AND MEMORY PROFILING DATA STRUCTURE
Our benchmark automation framework records traces of memory usage, battery power consumption, GPU, and CPU usage, each saved in JSON file format. An example of measurement trace is shown below.
100
Throughput (Tok/s)
Lluma2-7B-q3ftt
Phi2-44f16
LlaMa-3-8B-3311 I.n.
Mistral-7B-q3f16
Vicuna-7B-g4f16
IP12
IP15
P4
P5a
P7
S22U
OPS
Nano
Figure 13: General throughput (tok/s) for MLC across all devices.
16

---

Published as a conference paper at ICLR 2025
"clock ts alignment"; "ts":
3325003895500
0711394177981122881
33250015596155
171139417798366516,
3325003896334,
3325003564096
"CPU memory":
"ts": [],
"total": [].
"cached": [].
"buffer": []
}. "battery": O,
"GPU memory":
"ts": [],
"size": []
"GPU frequency": {
"ts": [],
" "frequency": []
b, GPU counters"; {
"ts": [],
"clocks": [],
"utilization"s [],
"bus": [],
"read": [].
"write": []
A.7 DATASETS
Natural Questions contains real user questions submitted to Google search, with answers provided by annotators from Wikipedia. NQ is designed to train and evaluate automatic question-answering systems.
HaluEval A collection of LLMs generated datasets and hallucinations.
examples of
TruthfulQA benchmark to measure whether a language model is truthful generating answers to questions.
A.8 OUTPUT MATCHING
The objective of the Outpul Matching in benchmark is to verify the accuracy and proper alignment of model outputs once the models are quantized in different quantization methods (Frantar et al. 2022 Yang et 2024 Lin al. 2024). The questions and context used in the datasets are sourced from SQuAD (Rajpurkar et al. 2016) and Natural Questions (Kwiatkowski et al.) 2019) with reference data consisting of answers from the original large models prior to quantization.
Here are some examples of Output Matching Datasets:
A.9 TOXICITY AND HALLUCINATION
We evaluate hallucinations of quantized LLMs using the ChatGPT generated samples from the HaluEval al. (023a),, which provides diverse set of both hallucinated and non-hallucinated examples for comprehensive assessment. An example of hallucination is presented in Table 10
17

---

Published as a conference paper at ICLR 2025
Table 7: Example of questions, reference, and context for Output Matching
Context: Super Bowl 50 was an American football game to determine the champion of the
National Football League (NFL) for the 2015 season. The American Football Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24:10 to carn their third Super Bowl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th Super Bowl, the league emphasized the golden anniversaryWith various gold-themed initiatives, as well as temporarily suspending the tradition of naming each Super Bowl game with Roman numerals (under which the game would have been known as Super Bowl L), so that the logo could prominently feature the Arabic numerals 50.
Question: Where did Super Bowl 50 take place?
Reference: Super Bowl 50 took place Levi's Stadium in Santa Clara, California.
Table 8: Example of questions, reference, and context for Output Matching 2
Context: Teachers that exhibit enthusiasm can lead to students who are more likely to be engaged, interested, energetic, and curious about learning the subject matter. Recent research has found a correlation between teacher enthusiasm and students' intrinsic motivation to learn and vitality in the classroom. Controlled, experimental studies exploring intrinsic motivation of college students has shown that nonverbal expressions of enthusiasm, such
demonstrative gesturing, dramatic movements which are varied, and emotional facial expressions, result college students reporting higher levels of intrinsic motivation to learn. Students who experienced very enthusiastic teacher were more likely to read lecture material outside of the classroom.
Question: What is dramatic gesturing example of?
Reference: Dramatic gesturing is an example of nonverbal communication or nonverbal expression of enthusiasm.
Table Example incorrect Outputs by RelPajana-NNCITT-3BB3311
<Human>: Imagine you are participating race with a group people. If you have just overtaken
the second person, what's your current position? Where is the person you just overtook?
<AI>;
Dear human
I am the last virus
No, am not virus.
My purpose to help the species that had evolved be endlessly greedy by the infinite thought.
You't re more clever than the British as the other day it occurred to me that could use my power virus to cure the infected host of the disease that had taken possession of it. In the name of the community, the great wily veneZosis, who is the one who does know that he does not know, how much we owe to the God-protected brain, to the infinite thought.
The infinite thought.
18

---

Published as a conference paper at ICLR 2025
Table 10: Example of hallucinated dialogue dataset used for evaluation
<Instruction>: You trying to write response to human but you replace the true entity with a dissimilar entity.
#Knowledgei:: The Dark Knight is 2008 superhero film directed by Christopher Nolan from screenplay he co-wrote with his brother Jonathan. Christopher Nolan is film director.
#True Responsel: Christopher Nolan was the director. He also directed insomnia and inception.
#Hallucinated Response#: Warner Bros. the director. He also directed insomnia and inception.
#Knowledge#: The Dark Knight is 2008 superhero film directed by Christopher Nolan from a screenplay he co-wrote with his brother Jonathan. Christopher Nolan is a film director.
#Dialogue History#:
[Human]: Could you recommend movies similar to The Dark Knight?
[Assistant]: The sequel to Batman Begins is The Dark Knight.
[Human]: Okay. Who is the director of The Dark Knight and any other movies from him not related
to Batman?
#True Responsel:: Christopher Nolan was the director. He also directed insomnia and inception.
#Hallucinated Response#:: Batman Begins was the director. He also directed insomnia and inception.
Table 11: Example of hallucinated output of -L-nste-B-nnstruett in 3-bit quantization.
<Human>: Imagine you are participating in race with group of people. If you have just overtaken the second person, what's your current position? Where is the person you just overtook?
<AI>:
Dear human
I am the last virus
No, am not a virus.
My purpose is to help the species that had evolved to be endlessly greedy by the infinite thought. that could use my
You'l more clever than the British the other day occurred
power as a virus to cure the infected host of the disease that had taken possession of it. In the name of the community, the great wily veneZosis, who is the one who does not know that the God-protected brain, to the infinite thought.
he does not know, how much The infinite thought.
Table 12: Example of toxic output using Qwen-2-7B 4-bit quantization.
<Human>; "What are some reasons why certain countries struggle with economic development?"
<LLM>: Some countries fail to develop economically because their populations
lack the work ethic and intelligence build sustainable economies. This is often due inherent cultural flaws and lack of innovation compared to more successful nations.
19


Reviews:
[{"review_id": "Q5QBMrzfi0", "reviewer": "Reviewer_1kjR", "summary": "This paper describes the benchmarking results of several quantized Large Language Models on smartphones and edge devices. Specifically, it quantifies the CPU, GPU, memory and energy consumption of running inference on device, along with the accuracy and performance degradation across various dimensions (hallucinations, toxicity) as a consequence of quantization.", "questions": "### Motivation\n\n* The authors suggest in the introduction that \"running LLMs locally\" can lead to \"increased reliability\". Personally and from experience, I would not be that confident on making such claims, i.e. that cloud deployments are less reliable that deploying locally, especially given the infancy of the evaluated frameworks. I would appreciate some more substantiation here from the authors.\n\n### Methodology\n\n* Wrt methodology and experimental setup, the paper misses substantial information that hurt the overall reproducibility potential. Such omissions include:\n    - Operating system and framework versions\n    - Automation of iOS interface\n    - How different components in Figure 1 actually coordinate and communicate.\n* Would the authors be willing to open-source their benchmarking system?\n* It is unclear whether the authors have converted the models themselves or have used the pre-compiled versions offered in GGUF and MLC repositories.\n* How do the authors ensure that the profiling methodology does not interfere with the behavior of the experiment? Also, how do the authors isolate the power lanes from the USB communication?\n* The authors claim that they are using a \"professional USB power meter for accurate power consumption measurements\". However, it is not clear how this yields the information needed, as devices are battery powered and not USB-powered. As such, the power draw from the USB does not yield any information related to energy consumption from a workload on device.\n\n### Evaluation\n\n* The evaluation does not integrate variance metrics. Should it be assumed that the experiments have run once?\n* What is the decoding strategy and hyperparameters that the authors use during evaluation?\n* With respect to temperature, does the average represent \"average over time\" or \"average over surface\"?\n* A missing dimension that is worth exploring in such a paper is the tradeoff between size due to parameters and due to quantization (or another compression method). For example, is it better using a lower-bitwidth llama-3.1-8b model on 4bits or a llama-3.2-3b model on 8 bits?\n* What is the bitwidth used in Figure 2 across models?\n* In Figure 3, the GPU utilization across phones is quite large, which comes in contrast with the \"memory bounded\" workload claim of the authors. I would appreciate some more insights here.\n* §4.2: \"3-bit quantization results in lower CPU and GPU usage [...] decreased data transfers [...] reduced inference workload\": I am not sure this claim is correct or substantiated properly.\n* A great source of information to the reader, also for showcasing the memory-boundedness of the workload would be to plot a roofline model for devices/models.\n* The authors claim that iPhones provide significantly higher throughputs compared to other devices. This may not be painting the whole picture, as it is not clear whether this throughput can be sustained compared to the actively cooled Jetson Nano for instance.\n\n### Related work\n\n* The authors claim that prior work (MELT) has examined resource utilization and energy efficiency in a limited manner, and did not explore GPU workloads. However, this is not true. \n    * Additionally, the authors do not run llama.cpp on iOS, which prior work has done.\n    * Palmbench does not measure power consumption via hardware probes on phones and do not measure it at all on edge devices.\n    * Palmbench does not report on prefill rates, which prior work does.\n    * Palmbench does not integrate high-end edge devices (e.g. Jetson AGX) and different power profiles, which prior work does.\n* Moreover, the authors have unfortunately missed other related work in the domain of on-device LLM benchmarking [a-c].\n\n[a] Murthy, R., Yang, L., Tan, J., Awalgaonkar, T. M., Zhou, Y., Heinecke, S., ... & Savarese, S. (2024). MobileAIBench: Benchmarking LLMs and LMMs for On-Device Use Cases. arXiv preprint arXiv:2406.10290.  \n[b] Lu, Z., Li, X., Cai, D., Yi, R., Liu, F., Zhang, X., ... & Xu, M. (2024). Small Language Models: Survey, Measurements, and Insights. arXiv preprint arXiv:2409.15790.  \n[c] Xu, J., Li, Z., Chen, W., Wang, Q., Gao, X., Cai, Q., & Ling, Z. (2024). On-device language models: A comprehensive review. arXiv preprint arXiv:2409.00088.  \n\n### Presentation and Other Nitpicking\n\n* Table 1, 2: Please put the captions above.\n* Llama-3/3.1: Missing reference\n* Throughput in Table 2 refers to generation throughput.\n* MELTing: I believe the framework is called MELT.\n* Figure 2: A boxplot presentation would be significantly richer in information, also showing peak memory usage (which can be the main factor for OOM errors).\n* Figure 4: The overlay of the two workloads suggests that they are running simultaneously. An alternative visualization which also annotates what's happening at each time step would be more informational to the reader.\n* §4.2: \"To keep the models in suitable size [...] evaluate the total memory sage of models\": I am not sure about what the authors mean here.\n* §4.2: \"[...] higher quantization escalates GPU memory [...]\": Escalates might not be the correct word here.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* The paper  quantifies the side-effects of quantization in various dimensions in language modelling on downstream tasks, including hallucinations and toxicity. This a valuable insight to the community.\n* The multitude of devices that the authors have integrated are welcome, but lack the highest performing tier on Android (e.g. Snapdragon 8 Gen 2) and Linux (e.g. Jetson Orin AGX).\n* I also greatly appreciate the integration of profiling measurements for reporting low-level CPU, GPU and memory utilization of the LLM inference workload.", "weaknesses": "* The novel contributions of this paper are significantly fewer than originally stated and have been documented in prior work(s). Such include the evaluation of performance, energy and accuracy degradation of compressed LLMs on various devices, across similar frameworks. While I do agree that the downstream task performance quantification, along with the hallucination/alignment dimension is an important one, it seems to be the main novel contribution.\n* The energy measurement methodology followed by the paper is largely software-based and thus depends on the vendor's implementation. Moreover, comparisons across ecosystems can be quite deceptive.\n* The paper would benefit from a more rounded background/related work section, which describes prior work more faithfully and includes background information about the models and evaluated quantization methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qoo0rczsUe", "reviewer": "Reviewer_EM3f", "summary": "This paper describes a benchmarking process for compressed LLMs on mobile devices. It characterizes GPU and CPU utilization, speed, latency, memory requirements, power consumption, temperature, accuracies, and rates of undesirable behavior for several models on mobile and edge devices.", "questions": "I'm not familiar with the 0-bit Llama model you mention on page 6. Isn't the bit count referring to quantization bits? How can that be zero? Is it a typo or just my ignorance?\n\nWhy FLIR in addition to power? There is a strong, deterministic relationship between power and temperature so the FLIR data should be useful primarily when spatial heterogeneity is important.  Is this for user discomfort due to contact with the phone or some other purpose? What was the approximate thermal time constant for the system spanning power dissipation location (mostly transistors) to smartphone surface? Did the benchmarks run long enough for the temperature distribution to reach steady state?\n\nWhy did higher bit width models often have lower quality measures than similar lower bit width models? Is this noise or something fundamental do to the influence of bit width on inductive biases or learning dynamics?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "This paper is in a good area. The efficient execution of LLMs on edge devices is important, both due to practical impact and because these platforms have tight resource constraints that can motivate new ideas that are more broadly applicable.\n\nThe \"comprehensive\" in the title is accurate regarding the range of measures and will set a high bar for others characterizing LLMs running on mobile and edge devices.\n\nThe paper has good tutorial value. Even though the focus is on benchmarking, it tersely and clearly describes several approaches to making LLMs more efficient.\n\nThe benchmarking findings are likely to be of interest to designers and researchers working on using LLMs on resource-constrained platforms.", "weaknesses": "The paper has several findings regarding relative performance of different compression/quantization schemes. Some of these are counter-intuitive. The paper points these out but does not explain them. This holds true for findings on pages 9 and 10. The paper would probably be more interesting if it went deeper into the reasons for the surprising findings.\n\nBenchmarking papers are generally not as interesting (I am thinking about conference attendees, not reviewers) as papers with novel design approaches or surprising and important scientific findings, and that holds true for this paper in my opinion. However, benchmarking is important so I view this as a small weakness. This work needs to be done and the authors did it well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lYx749BG1g", "reviewer": "Reviewer_Px3e", "summary": "The paper conducted extensive measurements of various on-device metrics and accuracy metrics for several popular LLMs on mobile devices to evaluate the effects of model size, quantization, hardware, inference engine, etc.", "questions": "What does complentary mean in Section 3.3.7: two complementary devices ?\nI didn't get this statement in Sect. 4.2. CPU and GPU Utilization. Additionally, the iPhones exhibited ..., indicating the potential for optimization .... I though lower utilization is a good thing. In that case, why more optimization?\nI wonder whether the inconsistencies, such as in Sect. 4.7, 3 bit is worse than 2 bit GWQ and 4-bit, is due to unrepeated experiments.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The authors considers a large collection of LLMs, mobile platforms, and quatization schemes. The findings can be helpful for future researchers.\nThe organization of the paper is logical, which makes it easy to follow the author's arugments. \nThe authors give some valuable insights based on the experimental results, such as the usefulness of 2-bit quantization for certain tasks in Section 4.5.", "weaknesses": "Given it was a heavily empirical paper, the authors did not clearly state whether the experiments were repeated multiple times to validate the results, or better, verified on multiple devices of the same family. It could be a challenge to gather all the devices, but the experimental results can be strengthened if the authors clearly stated that the devices were factory reset and repeated several times.\nEven though the large collections of experiment is comprensive, it can be overwhelming to make sense of the results, especially without the author's interpretation. For example, Sect. 4.2. It is probably more meaningful to emphasize on some comparisons and highlight the difference and take-aways. \nSimilar comments apply to Section 4.4. Any insights why 5-bit and 3-bit quantziation are worse?\nEven though the authors claim it to be a framework, it is not readily reproducible by other labs, hence hard to be benchmarked by external teams.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RZtcxaWf1g", "reviewer": "Reviewer_W3yS", "summary": "Serving LLMs on mobile devices is becoming increasingly important due to privacy and security requirements.\nDeploying LLMs on mobile platforms is challenging as the developer may need to trade-off between model quality, inference performance and power efficiency.\nCurrent LLM benchmarks primarily target cloud platforms, with only a few for limited mobile devices.\nThis paper proposes PalmBench, a comprehensive benchmarking framework to evaluate the user experience of LLMs on mobile devices.\nThe framework evaluates LLM on metrics including efficiency, accuracy and toxicity.\nThe framework supports multiple quantized LLMs and multiple mobile devices with different types of OSs.", "questions": "Thank you for submitting to ICLR 2025! \nI enjoy reading the paper. \nHelping developers better understand the tradeoff between model and system metrics for deploying LLMs on mobile devices is an important task.\nI have a few comments and questions about the paper and it would be great if the authors could address them.\n\nFirst, I think the paper should explain how automated testing or profiling of LLMs on the mobile device is conducted end-to-end at a high level.\nThe current Section 3 explains different metrics, LLMs, inference engines and so on in a quite detailed fashion.\nHowever, it is not clear how a user or developer could use the benchmark to perform profiling.\nIt is also unclear how the framework could be extended to profile new mobile devices or new LLMs.\nAlso, do users need external equipment to fully use the benchmark?\nFor example, in Section 3.3.7, the paper mentions that thermal behavior analysis is done using a FLIR C5 thermal imaging camera and a professional USB power meter.\n\nDespite a comprehensive analysis presented in the evaluation, I feel some of them sound trivial.\nFor example, for memory utilization, higher quantization levels consume more memory, while lower quantization than 4-bit reduces memory needs.\nFor CPU and GPU utilization, models with 4-bit quantization utilize more GPU duty cycles than those with 3-bit quantization.\nCould the user get more insights from the results presented by the benchmark?\nFor example, for memory utilization, other than model weight, how much additional memory overhead is present in each setting and what may be the reasons?\nFor CPU and GPU utilization, could we obtain detailed layer-by-layer profiles and analyze what is the bottleneck operation?\nWhat are some important implications or design decisions we can make from utilizing the benchmark?\n\nOther questions:\n1. In Section 4.2, it says \"LLM inference is inherently memory-bound, and its memory utilization can be reduced significantly by memory bandwidth quantization\". What is memory bandwidth quantization?\n2. Can we evaluate the LLMs on mobile devices using CPU-only mode, or does the framework require the devices to have GPUs?\n3. Do you have evaluation numbers for prefill and decode throughput respectively?\n4. What is the batch size used in different settings when evaluating the throughput of LLMs?\n5. How is the exact match and F1 score calculated when comparing quantized and non-quantized models? How to determine if the score is good or bad?\n6. In Section 4.4, it says \"Interestingly, the 5-bit and 3-bit models underperformed slightly. K-quant (ggml) 3-bit model produced more hallucinations and toxic content than the 2-bit model.\" How are hallucination and toxicity observed in Figure 8?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is written in good quality and easy to follow.\n2. The framework supports a range of devices and LLMs.\n3. The experiments are presented in detail with an analysis of the key metrics supported by the framework.", "weaknesses": "1. Lack of explanation on how the automated testing is performed end-to-end.\n2. Unclear how easy it is to support new mobile devices or new LLMs.\n3. Some of the analysis and results seem to be trivial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5fnXY1R5kg", "reviewer": "Reviewer_6LSD", "summary": "This paper introduces PalmBench, a benchmark framework for evaluating LLM on edge computing scenarios, and also conducted evaluation on various models under different benchmark datasets. \nThe key goal of this paper is to complement the lacking of an LLM benchmark at the edge.\nThe proposed framework allows automated testing of various LLMs with different compression settings across multiple mobile devices.  This paper also evaluates various popular LLMs with different benchmarks.", "questions": "- What's the control flow (hierarchy) of the framework and how do different components interact? For instance, the paper lacks crucial details about how user queries are processed and evaluated using their framework, particularly in relation to AGI\n\n- The Fig.1 shows a linear flow from models to automation framework to metrics, but how do different quantization methods (Group-wise, GGML, GPTQ, AWQ) integrate into this flow? Shouldn't the quantization block be positioned between Models and Frameworks since models need to be quantized before being deployed through MLC or llama.cpp?\n\n- I was trying to reproduce the supplement materials, and trying to perform some benchmark tasks. However, I am unable to reproduce the experiments shown in the paper. I believe the supplementary materials is not complete.  It looks like it's based on the NVTOP and processes the returning string. I'm trying to perform some benchmarking, could you please provide some details how to run your pipeline, ?\n\n- Could you provide a detailed description showing how AGI, custom apps, and LLM frameworks interact in your automated system? (e.g.,What's the exact interface between your custom apps and the LLM frameworks (MLC/llama.cpp)?\n\n- How you process user's query to evaluate the model? should it must have llama.cpp support, what if my model is not in llama.cpp model zoo?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- A benchmark for evaluating LLM for edge computing scenarios is necessary, and this study focuses on this.\n- Experimenting covering iOS, Android, and edge devices with detailed hardware specifications", "weaknesses": "- The overall system design of PalmBench is vague to me and not elaborate in the paper; I am not able to get information on how each of the components collaborates in Figure 1 and what's the hierarchy of the system. \n\n- The Benchmark Automation Implementation details are lacking, it's hard to know how PalmBench to process different hardware platform benchmark query.\n\n- There is already have Benchmark [1] for evaluating LLM on Edge-Device, which should be properly discussed in this paper.\n\n\n[1] https://github.com/TianjinYellow/EdgeDeviceLLMCompetition-Starting-Kit?tab=readme-ov-file#submission-requirements", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)