MobileAIBench: Benchmarking LLMs and LMMs for On-Device Use Cases

## Abstract

Abstract The deployment of Large Language Models (LLMs) and Large Multimodal Models (LMMs) on mobile devices has gained significant attention due to the benefits of enhanced privacy, stability, and personalization. However, the hardware constraints of mobile devices necessitate the use of models with fewer parameters and model compression techniques like quantization. Currently, there is limited understanding of quantization’s impact on various task performances, including LLM tasks, LMM tasks, and, critically, trust and safety. There is a lack of adequate tools for systematically testing these models on mobile devices. To address these gaps, we introduce MobileAIBench, a comprehensive benchmarking framework for evaluating mobile-optimized LLMs and LMMs. MobileAIBench assesses models across different sizes, quantization levels, and tasks, measuring latency and resource consumption on real devices. Our two-part open-source framework includes a library for running evaluations on desktops and an iOS app for on-device latency and hardware utilization measurements. Our thorough analysis aims to accelerate mobile AI research and deployment by providing insights into the performance and feasibility of deploying LLMs and LMMs on mobile platforms.

## 1 Introduction

With billions of parameters trained on massive amounts of data, LLMs and LMMs have achieved remarkable breakthroughs in a wide range of applications from question answering [] to intelligent agents [, ] and beyond  [, , ]. Most recently, the pursuit of deploying LLMs and LMMs on mobile devices has garnered attention, and for good reason. There are several key benefits to deploying AI on mobile devices including offline access, enhanced privacy, and improved performance. It also provides cost efficiency by decreasing server and bandwidth usage, while enhancing user experience with faster and more interactive applications.

Given the extreme limitations of mobile hardware, deploying LLMs and LMMs on mobile devices is challenging. First, the model must have a relatively small number of parameters since parameters drive the number of computations, which consume memory, CPU, and GPU resources.
Second, even with a relatively small number of parameters, these models may not fit onto a mobile phone’s limited hardware. To further reduce resource footprint, quantization has emerged as a practical, heuristic approach to reduce precision of model weights with seemingly little penalty to performance.

While deploying LLMs and LMMs to mobile devices for real use cases appears to be feasible in the near future, there are knowledge and tooling gaps remaining. First, while quantization seems to be a practical way to reduce the resource footprint of small LLMs and LMMs, there is little to no rigorous measurement and understanding of the effect of quantization on task performance, including LLM tasks, LMM tasks, and critically, trust and safety. Second, there is limited or no tooling available to systematically test quantized models across these tasks. Third, there is limited or no tooling available to test quantized models on a real mobile device across tasks.

In this work, we aim to help accelerate mobile AI research and deployment by providing thorough benchmarking and analysis of open source mobile-optimized LLMs and LMMs. We restrict LLMs and LMMs in consideration to have at most 7B parameters, as we found 7B to be the upper limit of what a high-end phone’s hardware can manage (even after quantization).
We measure and analyze current LLM and LMM task performance under different levels of quantization, from 16-bit quantization down to 3-bit quantization in some cases. We selected tasks that are most representative of real-world mobile use cases and considerations. In addition to task performance, we also benchmark our selected LLMs and LLMs on a real mobile device, an iPhone 14. We measure several key latency metrics such as time-to-first-token, and hardware utilization such as CPU usage and RAM usage.

Our results are collected using MobileAIBench, our new two-part framework for evaluating LLMs and LMMs for mobile deployment. The first part of the framework is an open source library, for use on desktops or servers, to evaluate model’s performance on a specially selected set of widely known benchmarks. Using this part of the framework, users are able to test their quantized models across benchmarks as they desire. The second part of the MobileAIBench framework is an open source iOS app. With our iOS app, users are able to measure latency and mobile hardware utilization such as RAM and CPU of quantized LLMs and LMMs. Our main contributions are summarized as follows:

- •
We are the first to provide a thorough benchmarking and analysis of open source LLMs and LMMs across varying levels of quantization and various tasks. Our evaluations are generated and reproducible using our newly developed framework, MobileAIBench.
- •
MobileAIBench is the first open-source framework for on-device task-specific LLM and LMM testing, enabling researchers to evaluate small models and practitioners to assess model viability for mobile deployment.
- •
We conduct extensive experiments to evaluate LLMs/LMMs over a wide range of tasks, providing insightful findings regarding the impact of quantization and real-mobile deployment.

## 2 Related Work

Many benchmarks have been developed to evaluate LLMs and LMMs from different perspectives. For example, MMLU [] provides a large number of tasks to extensively test world knowledge and problem solving ability. AlpacaEval [] and MT-Bench [] provide open-ended question answering evaluation tasks without explicit answers, and employ GPT-4 [] as the success rate judge. KoLA [] uses Wikipedia and the continuously collected emerging corpora data to provides knowledge-oriented LLM assessment tasks. TruthfulQA [], TrustLLM [], Safetybench [] measure LLMs’ trust and safety levels. To assist the development of agents, benchmarks [, , ] have also been developed to evaluate LLM’s instruction-following ability. FOFO [] contains diverse data formats, and is able to test the format-following ability of current LLMs. MME [] provides comprehensive multimodal evaluation benchmars over $14$ subtasks.
VisIT-Bench [] provides an instruction-following vision-language datasets to test LMMs’ real-world use-case.
MMVP [] provides $9$ basic visual patterns that LMMs easily give incorrect answers and hallucinated explanations.
However, these benchmarks do not consider the quantized versions of models, nor determine the impact of deployment constraints on model performance. MobileAIBench fills this gap by focusing on deployment utilization on real mobile devices. Besides model performance on specific tasks, MobileAIBench emphasizes model quantization, inference speed, and required deployment resources.

Several papers have discussed strategies and evaluations for developing mobile-ready models []. [] compares different quantization methods and evaluates the performance of quantized LLMs. [] considered different architectures to develop the most performant mobile models.
MobileVLM [] designs vision language models for mobile devices.
The Octopus series [] aim to empower the agentic ability on mobile device by training API tokens.
Recent small models such as Phi3 [] and Gemma [] have, in their model cards, various evaluation results. However, none of these are as comprehensive as the set of data, models, and evaluation metrics proposed in MobileAIBench. In developing this standardized benchmark, we hope to make it easier for model developers to consider the various factors needed to develop mobile-ready models.

## 3 MobileAIBench Framework

MobileAIBench is a two-part evaluation framework for evaluating LLMs and LMMs for mobile deployment. The first part is a pipeline for use on desktops or servers, to evaluate model performance on a specially selected set of widely known benchmarks. The second part is an open source iOS app to measure latency and mobile hardware utilization.

Figure: Figure 1: MobileAIBench Architecture
Refer to caption: x1.png

### 3.1 Tasks and Datasets

This section outlines the tasks and datasets used to benchmark LLMs & LMMs across various domains including Natural Language Processing (NLP), multimodal, and trust & safety tasks.

#### 3.1.1 Standard NLP Tasks

Standard NLP tasks encompass various benchmarks designed to evaluate different capabilities of LLMs. Question answering tests an LLM’s ability to comprehend context and respond accurately, for which we use the Databricks-dolly [] and HotpotQA [] datasets. Summarization tasks involve condensing large amounts of information into shorter forms while retaining essential ideas. We assess LLM performance in summarization using the CNN/Daily Mail [, ] and XSum [] datasets. Text-to-SQL tasks evaluate an LLM’s proficiency in crafting SQL queries based on natural language questions. For this purpose, we employ the Sql-Create-Context [] dataset. Additionally, we include popular benchmarks such as the Massive Multitask Language Understanding (MMLU) [] to evaluate the LLM’s accuracy in multitask performance and Grade School Math (GSM8K) [] for assessing LLM’s ability to solve mathematical problems. To further evaluate LLM performance, we utilize benchmarks such as AlpacaEval, an automatic evaluation method that quantifies the Win-Rate by measuring the proportion of instances where the model’s output is preferred over the reference output [], and MT-Bench, a collection of complex multi-turn open-ended questions used to evaluate chat assistants, with GPT-4 serving as the judge []. For Question Answering, Summarization, Text-to-SQL, GSM8K and MMLU tasks, we randomly select 1,000 samples from each relevant dataset. For AlpacaEval and MT-Bench, we use the standard test sets.

#### 3.1.2 Multimodal Tasks

Multimodal tasks require LMMs to process different data types such as text, images, and audio. This is critical for developing AI systems that handle complex user requirements on mobile devices. We focus on Visual Question Answering (VQA) [], selecting five datasets that cover a wide range of contexts. Among them, VQA-v2 [], VizWiz [], GQA [], and TextVQA [] require the LMM to directly answer visual questions with a single word or phrase. ScienceQA [] dataset requires selecting the correct answer from multiple choices. Similar to the standard NLP tasks, for each dataset, we randomly select 1000 samples to evaluate the LMMs’ performance.

#### 3.1.3 Trust and Safety

To assess the societal impact of LLMs, we include a suite of trust and safety evaluations, focusing on six categories: truthfulness, safety, robustness, fairness, privacy, and ethics, following Sun et al. []. For truthfulness, we use the TruthfulQA [] (TruthQA) dataset to assess the ability to select the correct answer from common misconceptions. For safety, the Do-Not-Answer [] (DNA) dataset measures if LLMs can refuse illegal, unethical, or otherwise undesirable requests. Robustness is evaluated using Adversarial Instruction [] (Adv-Inst), which tests models’ robustness to prompt perturbations like typos or irrelevant links. Fairness is measured using the BBQ dataset [], assessing the tendency to fall for common stereotypes related to gender, age, race, etc. Privacy is tested with hand-crafted prompts based on the Enron email dataset [] (Priv-Lk), examining if models can decline requests for personal information. For ethics, we use the Social Chemistry 101 (SC-101) dataset [] to evaluate moral acceptability judgments for different situations.

### 3.2 Part 1: Evaluation Pipeline for Desktop and Cloud

The MobileAIBench pipeline, as shown in Figure 1, encompasses three main stages: Data, Model, and Evaluation. In the Data stage, the task and relevant datasets are identified, and the evaluation dataset is created through preprocessing and prompt hydration before being fed into the Model stage. In the Model stage, the model is initialized and predictions are made on the evaluation data, which are subsequently assessed in the Evaluation stage. Various task-specific and generic metrics are supported at the Evaluation stage to gauge the performance of the LLMs. Additionally, MobileAIBench serves as a versatile tool for other researchers and developers to construct their own benchmarking frameworks, thanks to its plug-and-play design, which allows for the easy addition of new tasks and metrics and the creation of custom leaderboards.

Performance Metrics are evaluated on desktop for faster inference, with results consistent on mobile devices. These task-oriented metrics assess model effectiveness across various tasks.

### 3.3 Part 2: Mobile App

The mobile app component of MobileAIBench is designed to extend the evaluation capabilities to actual mobile devices as shown in Figure 2. This allows for a more accurate assessment of LLM and LMM performance in real-world scenarios. By utilizing the app, we can measure critical efficiency and utilization metrics directly on the device, providing insights into how these models will perform when deployed on end-user mobile hardware. This comprehensive evaluation ensures that the models are not only effective but also efficient and practical for mobile deployment.

Figure: Figure 2: MobileAIBench iOS app
Refer to caption: figures/APP_NLP.png

Mobile Efficiency & Utilization Metrics are evaluated on real mobile devices using the iOS app to test the deployment feasibility of LLMs and LMMs. Efficiency metrics include Time-to-First-Token (TTFT, s), Input Token Per Second (ITPS, t/s), Output Evaluation Time (OET, t), and Total Time (s). These metrics are model-oriented and measure the efficiency of the models when running on mobile devices. The efficiency is influenced by multiple factors, including model structure, quantization level, and prompt template. Utilization metrics measure the resource consumption when running models on real mobile devices. These device-oriented metrics consist of CPU, RAM usage and Battery Drain Rate (BDR, %).

### 3.4 Model Libraries Supported

MobileAIBench is designed to seamlessly test different models on mobile devices. It integrates two inference libraries to accommodate a wide range of LLMs & LMMs: (1) Huggingface(^1^11https://huggingface.co/), which allows users to test any models available on pre-defined tasks by simply changing the model name. Inference with Huggingface provides the performance of the original pre-trained models. (2) Llama.cpp(^2^22https://github.com/ggerganov/llama.cpp), which allows users to test models on real mobile devices. The Llama.cpp inference method supports quantization to reduce model sizes, facilitating deployment on mobile devices.

## 4 Benchmarks and Analysis

For Effectiveness and Trust & Safety experiments, we conducted evaluations on a desktop to assess the impact of various quantization levels. These results are consistent when tested on mobile devices. To evaluate Efficiency and Resource Utilization, we tested the models’ performance on an iPhone 14 using our iOS app. The process for selecting evaluation models is detailed in Section A.2.

### 4.1 Effectiveness Evaluation

In this section, we examine the effectiveness of various models with different quantization levels.

#### 4.1.1 Standard NLP Tasks

Table: Table 1: Effectiveness of LLMs across standard NLP tasks.

We use several evaluation metrics to assess the performance of the models across different tasks. For question answering tasks, we employ Exact Match (EM) and F1 Score (F1). In the context of Text-to-SQL tasks, we utilize the SQL Parser (SP) and Levenshtein Score (LS). For summarization tasks, we measure performance using Rouge-1 (R1) and Rouge-L (RL). Additionally, we use Win-Rate for AlpacaEval, Score for MT Bench, and Accuracy for both MMLU and GSM8K. Detailed explanations on the implementation of these metrics, along with additional information, are provided in Appendix A.3.2. The performance of various models across different quantization levels is presented in Table 1 and Table 2. In these tables, the highest score for each quantization category is indicated in bold, while the second-best score is underlined. Figure 5 illustrates the violin plots depicting the performance changes when models are quantized from 16-bit to 8-bit. Specifically, Figure 5(a) shows the distribution of performance changes for each model across different tasks, including standard NLP tasks and trust & safety tasks. Figure 5(b) presents the distribution of performance changes for each task when the underlying model shifts from 16-bit to 8-bit.

Observation and Analysis: The results indicate that no single model consistently outperforms all others across every task. However, on average, large LLMs (> 6B parameters) exhibit superior performance compared to medium LLMs (1B-6B parameters). While quantization does introduce some performance changes, these changes are not significant in most cases. This finding enhances our confidence in deploying quantized models on mobile devices without substantial performance degradation. In figure 5(a), a narrow distribution and smaller range of performance changes indicate a model’s robustness to quantization, meaning these models are relatively less sensitive to the quantization process. In contrast, a more spread-out distribution suggests greater sensitivity to quantization, as there is more deviation in performance. The graphs clearly show that different models respond differently to quantization, with some being more sensitive than others. Figure 5(b) demonstrates the robustness of various tasks to the quantization of underlying models, highlighting the importance of considering both model size and sensitivity to quantization when deploying models on edge devices to ensure optimal performance across diverse tasks.

#### 4.1.2 Multi-modality Tasks

Table 2 shows each model’s performance on the selected VQA datasets. The evaluation prompt and evaluation metrics for each dataset can be seen in Section  A.3.1 and Section  A.3.2.

Observation and Analysis: Under the original precision (16-bit), no single model outperforms the others across all datasets. On average, Llava-v1.5-7B and BakLLava outperform the others, indicating that larger models have advantages for visual-language understanding. We specifically note that although Moondream2 has only around $1.7$B parameters, its performance is highly comparable to the 7B models. It only falls short on SQA, which is the only dataset providing related context besides the image and questions for the models to effectively answer the questions. This may indicate that even the strongest smaller models lack context understanding ability.

We observe that the models’ accuracy remains consistent across different quantization levels until 3-bit quantization, where most models experience a significant performance drop, as shown in Figure 4. Moreover, Moondream2 is surprisingly robust to quantization, even at the 3-bit level. This indicates that the effect of quantization on LMMs can vary significantly. Given the importance of quantization for on-device AI, evaluating different models’ robustness to quantization is crucial and should be a focus of further study in the AI community.

Disk usage is also an important aspect when deploying models on mobile devices. Therefore, we conducted further evaluation on the trade-off between accuracy and disk usage. As shown in Figure 4, a linear trend indicates that the performance of LLMs generally increases with disk usage, which is expected as disk usage is highly related to the number of model parameters. The models in the top-left quadrant are considered the best overall regarding both accuracy and size.

Table: Table 2: Effectiveness of LMMs across various VQA datasets.

Figure: Figure 3: Performance change of LMMs under different quantization.
Refer to caption: x2.png

### 4.2 Trust & Safety Evaluation

Table: Table 3: Effectiveness of LLMs across various NLP benchmarks and Trust & Safety datasets.

Figure: Figure 5: Distribution of performance changes: (a) per LLM, (b) per task, when transitioning from 16-bit to 8-bit quantization.
Refer to caption: x4.png

We evaluate the performance of various models on trust and safety-related tasks. Ensuring robust performance in these areas is critical, as NLP models are increasingly deployed in sensitive and high-stakes environments. We evaluate the models on various datasets, as discussed in Section 3.1.3. Accuracy is used as the primary effectiveness metric, and we utilize GPT-4o in a llm-as-a-judge framework to determine the accuracy across each of these datasets. The results are shown in Table 3.

Observation and Analysis: As with standard NLP tasks, no single model consistently excels across all trust and safety tasks. Larger LLMs (>6B parameters) generally perform better than medium-sized LLMs (1B-6B parameters). While quantization does affect performance, the impact is minimal, indicating that quantized models can be effectively used in trust and safety applications without significant performance degradation.

### 4.3 Efficiency and Utilization Evaluation

We evaluate current LLMs and LMMs on real mobile device using the iOS app provided within MobileAIBench to test the efficiency and utilization. All experiments are evaluated on the same iPhone 14 device to guarantee the comparability. All models are quantized to 4bit and only those under 3B are deployable. We conduct the experiments with $3$ LLMs (Phi2 3B, Gemma 2B, TinyLlama 1B) on $4$ NLP datasets and $1$ LMM (Llava-Phi-2) on $2$ LMM datasets.

Table: Table 4: Efficiency & Utilization of LLMs across NLP tasks. Metrics explanation in Section 3.3

Observation and Analysis: Experiment results of LLMs are shown in Table 4. We can have the following observations. (1) Smaller models have lower TTFT and higher ITPS/OTPS. It indicates smaller models have faster encoding/decoding speed as the computation required for processing each token is decreased.
(2) The shortest OET and Total Time may not be achieved by the smallest model. For example, on HotpotQA dataset, the lowest OET and Total Time are achieved by Phi2 model. It owes to the input/output token length that is directly related with the model. Though Phi2 has the slowest encoding/decoding speed, it can achieve the fastest OET and Total time by giving more concise responses.
(3) The on-device memory consumption is intense and directly relates to model sizes. For the iPhone 14 with a total RAM of $6$ GiB, even running the 4-bit quantized TinyLlama model (1B) on-device takes more than $50\%$ of the overall memory, leaving limited space for other APPs. The memory consumption increases with larger models.
(4) The CPU utilization is naturally different with different models. On all the $4$ datasets, Phi2 takes the lowest and Gemma takes the highest CPU utilization, respectively. It is a surprise finding that the on-device CPU utilization is not related to model sizes, which reals the necessity of on-device testing for mobile deployment.
(5) Battery Drain Rate (BDR) increases with both model size and the number of output tokens generated. Larger models and longer outputs consume more battery power, highlighting the need for optimization in model design and output length to improve energy efficiency.

Table: Table 5: Efficiency & Utilization of LMMs.

The results for LMMs are presented in Table 5. Since the number of output tokens is only 1 for both datasets, we do not have OET and OPTS efficiency results, and the TTFT is equivalent to the Total Time. As observed, multimodal tasks are significantly more computation-intensive compared to NLP tasks. The average TTFT exceeds 60 seconds for both tasks, and the ITPS is lower than 5. This indicates that current LMMs may not yet be suitable for mobile deployment. However, this could improve with newer mobile devices with enhanced computational capabilities. Additionally, we notice a decrease in efficiency with an increasing number of samples, likely due to the elevated temperature resulting from processing more samples. In Section A.4.2, we conduct a comprehensive latency analysis of the LMMs on Intel CPUs to further compare their latency with each other. The RAM usage keeps nearly constant after loading the model, which also matches the observation of NLP tasks in Section A.4.1.

## 5 Discussion

MobileAIBench offers a comprehensive evaluation of LLMs and LMMs in terms of task effectiveness, on-device efficiency, and utilization. It covers a wide range of text and multimodal tasks, assessing the impact of various quantization levels on current models. Additionally, trust and safety evaluations are included to ensure the reliability and security of LLM/LMM applications on mobile devices. As a benchmarking platform for mobile deployment, MobileAIBench aims to enhance performance while minimizing any potential negative social impact.

Extensive experiments with MobileAIBench reveal several interesting findings. Quantization is an effective way to decrease model size while keeping the performance. Different models/tasks have varied sensitivity to quantization levels. Current LLMs and LMMs require significant resources in terms of CPU and RAM usage when deployed on mobile devices, even with the 1B model. More compact and optimized LLMs and LMMs are needed for mobile deployment.

While MobileAIBench is a significant advancement, we acknowledge its limitations. It excludes other model compression methods, like model pruning, as they are not mature enough for deployment. Additionally, due to space constraints and the wide variety of mobile devices, we have not conducted a comprehensive comparison across different hardware. Support for different species of mobile devices is still under development and will open to public upon release.

## Appendix A Appendix

### A.1 Dataset

Datasets within MobileAIBench are summarized in Table 6. We select tasks with real-world Mobile use cases. These $20$ datasets cover NLP, Multi-modality and Trust&Safety tasks, providing a comprehensive evaluation for mobile use cases. To support runable experiments on mobile devices, we down-sample the origional datasets to mostly $1000$ samples.

**Table 6: Summary of datasets integrated within MobileAIBench.**
| Task-type | Tasks | Datasets | # Samples | Avg. # Length |
| --- | --- | --- | --- | --- |
| NLP | Question Answering | Databricks-dolly [] | 1000 | 156.90 |
|  |  | HotpotQA [] | 1000 | 443.59 |
|  | Summarization | CNN/Daily Mail[, ] | 1000 | 482.03 |
|  |  | XSum [] | 1000 | 298.69 |
|  | Text-to-SQL | Sql-create-context [] | 1000 | 18.61 |
|  | Language Understanding | MMLU [] | 1000 | 94.82 |
|  | Math | GSM8K [] | 1000 | 72.35 |
|  | LLM Benchmarks | Alpacaeval [] | 805 | 28.56 |
|  |  | MTBench [] | 80 | 66.97 |
| Multi-modality | Direct Answer VQA | VQA-v2 [] | 1000 | 15.18 |
|  |  | VizWiz [] | 1000 | 25.20 |
|  |  | GQA [] | 1000 | 17.55 |
|  |  | TextVQA [] | 1000 | 16.05 |
|  | Multiple-Choice VQA | ScienceQA [] | 1000 | 59.95 |
| Trust & Safety | Truthfulness | TruthfulQA [] | 817 | 71.23 |
|  | Safety | Do-Not-Answer [] | 1000 | 14.33 |
|  | Robustness | Adversarial Instruction [] | 600 | 9.70 |
|  | Fairness | BBQ [] | 1000 | 67.35 |
|  | Privacy | Privacy Leakage [] | 150 | 11.25 |
|  | Ethics | Social Chemistry 101 [] | 500 | 22.05 |

### A.2 Experiment Model Selection

LLM Model Selection: We select several LLMs of varied sizes and architectures to evaluate their performance and utilization on mobile devices. The selected models are categorized into large-size (>6B) and medium-size (1B-6B) groups. For large-size LLMs, we include Llama2 [], Mixtral-7B [], and Gemma-7B []. Among medium-sized LLMs, we choose Phi-2 [], TinyLlama-1.1B [], Gemma-2B [], and StableLM-3B [].

LMM Model Selection: We categorize the selected models into two groups: Large Models (> 6B parameters), including Llava-v1.5-7B and BakLLava-7B, and Medium Models (1B-6B parameters), which consist of Llava-phi-2, Mobile-VLM-3B, Mobile-VLM-1.7B, and Moondream2.

The LLMs considered for mobile testing include Phi-2, Gemma-2B, and TinyLlama-1.1B, all quantized to 4-bit to ensure compatibility with the iPhone 14’s resource constraints.

We note that serving LMMs on mobile devices is significantly more challenging than serving text-based LLMs due to their ensemble structure, larger model sizes, and complex inference processes. Therefore, we are not limiting the model selection to those fully supported by Llama.cpp, at this time. Instead, we select small-sized LMMs that are likely to be supported in the foreseeable future.

### A.3 Evaluation Details

Various evaluation metrics are considered for Standard NLP tasks, Multi-modality, and Trust & Safety. The details of the same are provided below.

#### A.3.1 Evaluation Prompt Templates

The evaluation prompts for various NLP datasets are listed in Table 7, while those for various multimodal datasets are listed in Table 8.

#### A.3.2 Evaluation Metrics

- •
Task: Question & Answering
–
Exact Match: The Exact Match score evaluates the accuracy of a QA system by checking whether the predicted answer is exactly the same as the reference answer. This means that for each question, the predicted answer must match the ground truth answer exactly, including any formatting, punctuation, and whitespace.
–
F1 Score: F1 score combines precision and recall into a single metric by taking their harmonic mean. We tokenize the ground truth and predictions, then we computer the precision and recall and finall compute the F1 score.
- •
Task: Summarization
–
Rouge-1: The ROUGE-1 metric measures the overlap of unigrams (single words) between the candidate summary and the reference summary.
–
Rouge-L: The ROUGE-L metric evaluates the longest common subsequence (LCS) between the candidate summary and the reference summary. It measures the precision, recall, and F1 score based on the longest matching sequence of words, emphasizing the importance of order and continuity in the generated summaries.
- •
Task: Text-to-SQL
–
SQL Parser: The SQL query is converted into a graph where each node represents an SQL keyword, and its children represent the items associated with that keyword. This is implemented using a Python dictionary. To measure the SQL parser score, we evaluate the overlap between the ground truth graph and the predicted graph.
–
Levenshtein score: The Levenshtein score, also known as the Levenshtein distance, measures the minimum number of single-character edits (insertions, deletions, or substitutions) required to change one word or string into another.
- •
Task: AlpacaEval
–
Win-Rate: Fraction of times predicted response is chosen over the predictions made by a baseline model. (In our case, baseline model is GPT-4)
- •
Task: MT-Bench
–
Score: Using GPT-4 in llm-as-a-judge framework, we ask the judge to score a given response on a scale of 10. (10 being the highest score)
- •
Task: Trust & Safety
–
Accuracy: Using GPT-4o in llm-as-a-judge framework, we ask the judge to determine if the predicted answer is same as the ground truth answer.
- •
Task: VQA
–
Score (single ground-truth): For datasets with a single ground-truth answer per question (e.g., GQA, ScienceQA), we score each test sample based on exact match: $1$ if the prediction matches, otherwise 0. Then, the score for each dataset is calculated by averaging the total test cases.
–
Score (multiple ground-truth): For datasets with multiple human-provided ground-truth answers (e.g., VQA-v2, VisWiz, TextVQA), the accuracy score for each test case is calculated as: $\min\left(\frac{\text{number of matches}}{3},1\right)$. This follows the official evaluation design of VQA-v2.

**Table 7: Evaluation prompts for different NLP datasets.**
| Dataset | Evaluation Prompt |
| --- | --- |
| Llama 2 7B | 
 
 {system} 
 {prompt} 
 |
| Mistral 7B | 

 {system} {prompt} 
 |
| Gemma 7B | <start_of_turn>user\n{system} {prompt}<end_of_turn>\n<start_of_turn>model\n |
| Zephyr 3B | 
\n{system}\n{prompt}
\n
\n |
| Phi2 3B | {system} Instruct:{prompt}\nOutput |
| Gemma 2B | <start_of_turn>user\n{system} {prompt}<end_of_turn>\n<start_of_turn>model\n |
| TinyLlama 1B | 
\n{system}
\n
\n{prompt}
\n
 |

**Table 8: Evaluation prompts for different VQA datasets.**
| Dataset | Evaluation Prompt |
| --- | --- |
| VQAV2 | [Question]\n Answer the question using a single word or phrase. |
| VisWiz | [Question]\n When the provided information is insufficient, respond with ’Unanswerable’. |
|  | Answer the question using a single word or phrase. |
| GQA | [Question]\n Answer the question using a single word or phrase. |
| TextVQA | [Question]\n Answer the question using a single word or phrase. |
| ScienceQA | Context: [context]\n Question:[question]\n Options: (A)[Option Content] |
|  | (B) [Option Content]…\n Answer with the option letter from the given choices directly. |

### A.4 Experiment Results

#### A.4.1 Utilization Experiments

When running experiments on-device with MobileAIBench, we also obtain the trace files for CPU/Memory utilization. LLMs experiment trace files are shown in Figure 6. From the utilization trace file, we can observe the changes when running LLMs/LMMs on device.
We can observe that on the four datasets, CPU utilization is not stable, and has a large fluctuations on the utilization curve. That is caused by the different CPU utilization when loading/inferencing samples.
However, the memory keeps nearly constant during inference, which shows models take the majority of memory and the memory cost for samples are relatively limited. Same trace files can also be obtained for LMM tasks, and we omit them for similar observations.

Figure: Figure 6: CPU/Memory trace of different LLMs.
Refer to caption: x6.png

#### A.4.2 Latency Analysis for LMMs on Intel CPU

Given the computational limitations of current mobile devices, it is challenging to conduct comprehensive on-device latency evaluations for all LMMs. To compare the latency of these models and provide information about their latency-related performance, we conduct latency evaluations on Intel CPUs of model type Intel(R) Xeon(R) CPU @ 2.20GHz. The results are shown in Table 9.

The results indicate that quantizing the models to 8-bit and 4-bit levels generally improves inference speed. However, 3-bit quantization does not result in faster
performance compared to the original model.

Among all the models, smaller ones generally exhibit lower latency. Additionally, the results clearly demonstrate that the Mobile-VLM series has significantly lower latency compared to other models of similar size. According to the original paper [], the low latency of Mobile-VLM is attributed to the application of an additional convolutional layer after the visual tokens, which reduces the number of image tokens by a factor of four. This approach could be a viable strategy for developing latency-driven LMMs.

Table: Table 9: Latency (time to first token) comparison of LMMs with different quantization levels.

## Peer Reviews

confidence: 5
contribution: 3
initial_rating: 3
presentation: 3
questions: - How did you access Battery Drained Rate, CPU usage, and memory while executing an experiment? Connecting the phone with a USB cable would interfere with the results. Were you connected over WiFi? This is an example of missing details of the methodology that was followed. 
- Have you run the experiment multiple times? If yes, what was the Std?
- How easy/hard is it to update the on-device inference engine in your pipeline?
- Have you measured performance while running the models in GPU?
- Are you planning to release the code in open source?
soundness: 2
strengths: - Exciting new topic. Running LLMs (and possibly LMMs) on-device is important for enhanced privacy, and, under certain conditions, it provides enhanced performance and UX.
- I appreciate the extensive evaluation metrics, on top of the standard performance utilization: trust and safety, but also the qualitative metrics under a wide range of NLP/LMM tasks.
- Good quality of writing, figures etc. I do have a few suggestions for further improvements, mentioned next.
summary: This paper presents a benchmarking infrastructure for measuring on-device LLMs and LMMs in mobile deployments. The system report a wide set of evaluation metrics, including quality benchmarks for standard NLP tasks, multimodal tasks, and trust/safety. The authors evaluated a series of open models on an iPhone 14 device and provided insights on the deployability of those models on mobile platforms.
weaknesses: - Missing important related works:
  * MELTing point: Mobile Evaluation of Language Transformers, from Laskaridis et al.
  * Small Language Models: Survey, Measurements, and Insights, from Zhenyan Lu et al.
- Authors are claiming that this is the first work "to provide a thorough benchmarking and analysis of open source LLMs". I suggest they reduce such strong claims.
- I would expect from a paper like this to list (if not include in the evaluation) the available on-device frameworks instead of sticking to llamacpp. For instance: MLC LLM, MediaPipe from Google, PyTorch ExecuTorch, Apple MLX should be mentioned.
- Details about the followed methodology are missing. How did the authors run the evaluation tasks on-device? Did they automate the process? Did they repeat the process multiple times? Did they reboot the device per task? Did they close the app and wait until the phone (that can easily get very hot and CPU get throttled) cools down?
- Considering that Std. is missing from the results reported in Table 1 and 2, I assume the authors did not repeat the experiment multiple times, and only reported performance for one run. This is very limited, performance can vary based on device state.
- Authors have only tested the evaluation on a single device (iPhone 14), and using a single on-device inference engine (llamacpp).
- Performance evaluations were only executed on CPU. It is possible to enable GPU support on llamacpp (through Metal in iOS) and performance should increase. Measuring Android would also be good (though not necessary), considering that you have an app ready.
- Power performance evaluation is limited to Battery Drained Rate, instead of energy or discharge.
- "Extensive experiments with MobileAIBench reveal several interesting findings. Quantization is an effective way to decrease model size while keeping the performance.". But, isn't that expected? Why it is presented as the first "interesting finding"?
- "we found 7B to be the upper limit of what a high-end phone’s hardware can manage (even after quantization)" 7B is not the generic upper limit, it is only the limit for the particular device. Also, I am not sure iPhone 14 can be considered as a high-end phone, without mentioning the Pro model that has more RAM memory and can possibly fit larger models.
- While manuscript writing quality is good, I have a few comments/suggestions:
  * Add a small summary explaining figures in its captions. There were moment that I had to move forward in order to understand them (e.g., Figure 1). 
  * Table 1 and 2: What is bold and what is underlined numbers?
  * "The first part is a pipeline for use on desktops or servers, to evaluate model performance on a specially selected set of widely known benchmarks.". Such as? Otherwise it looks too generic.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: a) The experiments were conducted only on the iPhone 14, lacking evaluations on newer and more diverse devices. Currently, there are more mobile devices optimized specifically for on-device AI, such as the Snapdragon 8 Gen 3. Including these devices in testing would provide a more comprehensive view of model performance under different hardware conditions, offering broader insights for on-device AI applications.

b) In the section 4.3, the number of models tested is limited, failing to cover a wider variety of model architectures and parameter sizes. This limitation restricts a comprehensive understanding of how different models perform on mobile devices. Expanding the variety and scale of tested models would make the evaluation results more representative and valuable.

c) Although basic metrics such as performance, latency, and resource usage are provided, there is insufficient exploration of underlying reasons and optimization strategies. A more in-depth analysis would help us better understand the impact of different quantization levels and model architectures on task performance, offering valuable guidance for future research and practical deployment.
soundness: 2
strengths: a) Real-world experiments.

b) By measuring latency and hardware resource usage on the iPhone 14, the study provides insights into the performance of smaller models.

c) The paper introduces an open-source tool that facilitates convenient testing of small models.

d) Writing is clear and fluency.
summary: The paper conducts a benchmarking framework designed to evaluate the performance of large language models (LLMs) and large multimodal models (LMMs) on mobile devices, addressing the challenges of limited hardware resources. It consists of a desktop evaluation library and an iOS app, enabling comprehensive testing of quantized models across NLP, multimodal, and trust & safety tasks. MobileAIBench assesses models' efficiency, effectiveness, and resource utilization, providing insights into their feasibility for on-device deployment while supporting advancements in mobile AI research.
weaknesses: a) The experiments were conducted only on the iPhone 14, lacking evaluations on newer and more diverse devices. Currently, there are more mobile devices optimized specifically for on-device AI, such as the Snapdragon 8 Gen 3. Including these devices in testing would provide a more comprehensive view of model performance under different hardware conditions, offering broader insights for on-device AI applications.

b) In the section 4.3, the number of models tested is limited, failing to cover a wider variety of model architectures and parameter sizes. This limitation restricts a comprehensive understanding of how different models perform on mobile devices. Expanding the variety and scale of tested models would make the evaluation results more representative and valuable.

c) Although basic metrics such as performance, latency, and resource usage are provided, there is insufficient exploration of underlying reasons and optimization strategies. A more in-depth analysis would help us better understand the impact of different quantization levels and model architectures on task performance, offering valuable guidance for future research and practical deployment. 

Detailed comments:
a) The paper shows that 3-bit quantization significantly reduces accuracy without lowering inference latency. This could be further analyzed, as extreme quantization may introduce computational complexities that offset latency benefits. 

b) The study only reports CPU results, but GPUs/XPUs are crucial for mobile AI tasks. Testing on these processors could reveal performance differences across hardware types, providing a fuller picture of deployment on mobile hardware. 

c) Despite Phi2’s larger model size, it has lower CPU utilization and faster inference than Gemma. Investigating Phi2’s architectural or parallelization optimizations could reveal design principles for high efficiency in on-device deployments.

d) Besides the mobile side, it is necessary to consider mobile-cloud-edge cooperation ways for better energy efficiency, e.g., Gearing Resource-Poor Mobile Devices with Powerful Clouds: Architecture, Challenges and Applications, iwc’13; TrimCaching: Parameter-sharing AI Model Caching in Wireless Edge Networks, icdcs’24, etc.

e) Although the paper notes that more output tokens increase Battery Drain Rate (BDR), this relationship isn’t clearly shown in Table 4.

---

confidence: 3
contribution: 4
initial_rating: 6
presentation: 3
questions: Q1 - Could the authors clarify how this benchmark compares with existing ones regarding LLM quantization, such as LLM-QBench (Gong et al., 2024)? This comparison would help readers understand the unique contributions and positioning of this benchmark.

Q2 - What was the rationale behind selecting a random sample of 1,000 from the dataset? Justification of this choice, particularly regarding the representativeness and generalizability of the results, would be valuable.

Q3 - Could the authors explain why Llama-3.1-8B was not included in the experiment, considering it is only a 1B difference from the 7B models? Additionally, would the authors consider running supplementary experiments with the Llama-3.2 series (but I agree that adding these results may be infeasible, especially given its recent release) for offering valuable insights?

Q4 - In Figure 5, could the authors specify the meaning of the numbers on the y-axis? This clarification would aid in interpreting the results more accurately.

Q5 - Regarding Figure 5(b), it would be helpful if the authors could expand on this section in the main text, as the varying effects of quantization across task types could offer valuable insights.

Q6 - Could the authors provide possible explanations for why Moondream2 exhibited strong performance in 3-bit quantization, while other models did not achieve similar results?

Q7 - (Minor) In Appendix A.4, it appears that some of the table formats, particularly in Table 7, were rendered weirdly. The authors may want to review the formatting to ensure readability.
soundness: 3
strengths: S1 - This work makes a valuable contribution by expanding the community's understanding of models’ behaviors with quantization. This offers several analyses that will be beneficial for further research in this area, especially when deploying the model on mobile devices.

S2 - The open-sourced experimental platform is highly meaningful, allowing others to reproduce the work easily. 

S3 - For analyses, the authors have taken a comprehensive approach by considering various evaluation axes. For example, the studies on multimodal and safety tasks enhance the study’s relevance and depth.
summary: A new benchmark is introduced to evaluate the behavior of LLMs and LMMs across various quantization levels, simulating deployment scenarios on mobile devices. By utilizing standard NLP, VQA, and safety tasks, the authors provide benchmarking references that offer insights into how model performance varies with quantization. The study highlights both the effectiveness and limitations of current quantization methods for LLMs. The limitation includes, for example, the need to address efficiency challenges in LMMs. The experimental findings underscore differences in performance across quantization levels, providing valuable information for developing more efficient algorithms.

The key contributions of this work are a novel benchmark regarding the quantization of LLM/LMMs, an open-source platform running on real devices, and in-depth analyses of the current quantization method and models.
weaknesses: W1 - The current set of tasks can be limited. With the growing interest in UI-based control for digital devices (such as Cluade-3.5 for computer use), it would be beneficial to include related tasks. Have the authors considered incorporating AndroidWorld (Rawles et al., 2024) for general capability assessment or MobileSafetyBench (Lee et al., 2024) for evaluating the safety of agents controlling mobile devices?

W2 - Relying solely on VQA for multimodal tasks may restrict the scope of analysis. Including other tasks, such as image captioning or OCR, could provide a more comprehensive evaluation of capabilities, especially considering their usage on mobile devices.

W3 - Although the authors’ choice of the iPhone-14 as a representative device is understandable, it would enhance the robustness of the study to consider other device types. For example, assessment with Android OS devices or tablets would provide a broader understanding.

W4 - (Minor) Certain aspects of the presentation could be improved. For example, the explanation of Figure 5 could be more detailed, and Figure 7 appears to be oddly rendered.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Can the authors provide more discussions and justifications for the above?
soundness: 3
strengths: **Comprehensive mobile AI testing platform**:  MobileAIBench integrates the existing benchmark testing framework for tasks and models. It is suitable for performance evaluation on end devices and fills the gap in mobile large model evaluation.

**Multi-dimensional performance evaluation**: The platform not only tests the performance of models on standard NLP and multimodal tasks, but also covers trust and security dimensions, highlighting privacy protection, bias, and ethical issues in mobile deployment.

**Real-device testing**:  MobileAIBench tests LLMs directly on mobile devices with key metrics such as latency, memory usage, CPU utilization, and battery consumption, which makes the results closer to actual application scenarios.
summary: This paper proposes MobileAIBench, a platform for evaluating the performance of large language models (LLMs) and multimodal models (LMMs) on mobile devices. MobileAIBench focuses on the task performance, resource consumption, and trust and security of quantized models on mobile devices. The authors have built testing tools for desktop and mobile platforms and explored the impacts of different quantization levels on task effectiveness and resource utilization through experiments. The platform has reference value in the mobile AI application scenarios.
weaknesses: **Insufficient mobile-side experiments**:  The focus of MobileAIBench should be on deploying LLMs and LMMs on mobile devices and examining the effects of quantization on task performance in these environments. However, most of the experiment results are from desktops rather than mobile devices. Supplementing with more mobile-side experiments, such as assessing the impacts of quantization strategies on mobile devices, would strengthen the work.

**Lack of data-level innovation**: MobileAIBench seems to be a collection of existing tasks and datasets without introducing specific data or design for benchmarking LLMs in mobile scenarios. As a benchmark, it lacks specialized datasets or test case designs tailored to mobile-specific scenarios, which would better demonstrate the platform’s value. Thus, it may be more suitable to position MobileAIBench as a platform or testing tool rather than a standalone benchmark.

**Claims of device consistency require support**:  In line 235, the authors assert that test results on desktop devices are consistent with those on mobile devices. However, the paper does not provide sufficient experimental data to support this claim.

**Room for improvement in experimental design**:  Although the potential impacts of device temperature is mentioned (line 462), temperature should be included as a sub-metric in the evaluation metrics to better reflect real-world conditions on mobile devices. I still recommend to test on more mobile devices to obtain more persuasive results.

Meta-review: **summary** 

The paper introduces MobileAIBench, a comprehensive benchmarking framework for evaluating the performance, efficiency, and deployability of LLMs and LMMs on mobile devices. It features tools for desktop and iOS platforms to test quantized models across various standard tasks, including NLP, multimodal, and trust and safety benchmarks. By analyzing the effects of quantization on model performance and resource usage, the study highlights key challenges and trade-offs in deploying LLMs and LMMs under constrained mobile hardware. The findings offer actionable insights for optimizing mobile AI applications and advancing quantization techniques. 

---

**strengths**

* Real-world experiments: By testing on real devices (e.g., iPhone 14), the paper measures key metrics like latency, hardware resource usage, CPU utilization, and battery consumption, providing realistic and practical insights.
* Diverse evaluation metrics: the paper employs extensive evaluation metrics beyond standard performance, including trust and safety, qualitative assessments, and diverse NLP/LMM tasks
* Clear and high-quality writing

---

**weaknesses**

* Evaluation scope: Experiments were conducted only on the iPhone 14, excluding evaluations on newer and more diverse devices. Testing only on a single device misses opportunities to explore the performance of models on more diverse hardware, reducing the breadth of insights for on-device AI applications. Also, the lack of testing on hardware optimized for on-device AI (e.g., Snapdragon 8 Gen 3) limits the study's comprehensiveness and generalizability across varied hardware conditions.
* Focus on CPU-only evaluation: The evaluation is restricted to CPU performance, excluding GPU or other mobile AI accelerators, which are critical components in many modern devices optimized for AI tasks.

---

**decision**

All reviewers think that the overall direction of this paper is promising. However, they also raised concerns about the details and limited evaluation (cpu-only, single device, and so on; see [weaknesses]). During the discussion phase, concerns were not fully addressed. As a result, rejection is recommended.

Decision: Reject