LinguaLinked: Distributed Large Language Model Inference on Mobile Devices (2024)

Abstract: Deploying Large Language Models (LLMs) locally on mobile devices presents a significant challenge due to their extensive memory requirements. In this paper, we introduce LinguaLinked, a system for decentralized, distributed LLM inference on mobile devices. LinguaLinked enables collaborative execution of the inference task across multiple trusted devices and ensures data privacy by processing information locally. LinguaLinked uses three key strategies. First, an optimized model assignment technique segments LLMs and uses linear optimization to align segments with each device’s capabilities. Second, an optimized data transmission mechanism ensures efficient and structured data flow between model segments while also maintaining the integrity of the original model structure. Finally, LinguaLinked incorporates a runtime load balancer that actively monitors and redistributes tasks among mobile devices to prevent bottlenecks, enhancing the system’s overall efficiency and responsiveness. We demonstrate that LinguaLinked facilitates efficient LLM inference while maintaining consistent throughput and minimal latency through extensive testing across various mobile devices, from high-end to low-end Android devices.

heading: 1 Introduction
text: The past decade has witnessed a seismic shift in the machine learning (ML) landscape, particularly with the rise of large language models (LLMs), which are built atop transformer decoders (Vaswani et al., 2023). These LLMs (Brown et al., 2020; Kaplan et al., 2020, Hoffmann et al., 2022; Chowdhery et al., 2022; Zhang et al., 2022; Touvron et al., 2023; Workshop et al., 2023) have achieved stateof-art performance on Natural Language Processing (NLP) benchmarks such as text generation, question answering, machine translation, and text
0* Equally contributed.
summarization, and led to commercial offerings such as OpenAI ChatGPT and Github Copilot. Recent research has established that as the number of parameters in these models increases, they demonstrate enhanced capabilities in various language tasks (Alabdulmohsin et al., 2022; Clark et al., 2022; Huang et al., 2020; Patel and Pavlick, 2022, Hendrycks et al., 2021; Cobbe et al., 2021).
However, deploying these LLMs on mobile devices is challenging due to their significant memory and processing requirements. Traditional server-based inference raises privacy and bandwidth issues. An alternative is distributed inference, where LLMs are split into smaller segments across multiple devices, reducing the need for heavy model weight quantization and maintaining accuracy. While previous studies have looked into distributed model deployment on mobile computing platforms (Hu et al., 2019; Naveen et al., 2021; Zeng et al., 2021; Zhou et al., 2019), these have largely concentrated on smaller-scale models used in computer vision applications, which have a much smaller memory footprint compared to LLMs and
1 160
typically do not need iterative inference. In this paper, we present LinguaLinked1, a decentralized distributed inference system for LLM deployment on mobile devices. The core concept behind LinguaLinked is distributing model segments across ’trusted’ devices shown in Figure 1, such as personal smartphones and tablets. This approach overcomes the limitations of individual device capacities, privacy concerns, and bandwidth constraints. However, it faces challenges like managing diverse device capabilities, handling data dependencies between model segments, and adjusting to dynamic resource availability.
LinguaLinked addresses these challenges with three key components: optimized model assignment that aligns model segments with device capabilities while minimizing data transmission, runtime load balancing to redistribute tasks and prevent bottlenecks, and optimized communication to ensure efficient data exchange between model segments. We perform a thorough evaluation of LinguaLinked on high-end and low-end Android devices. In a single-threaded setting, compared to the baseline, LinguaLinked achieves an inference performance acceleration of approximately 1.11× to 1.61× across both quantized and fullprecision models. With multi-threading, the system exhibits further improvements, achieving acceleration rates of approximately 1.73× to 2.65× for both quantized and full-precision models. Runtime load balancing yields an overall inference acceleration of 1.29× to 1.32×. Importantly, our findings indicate that LinguaLinked’s performance
1https://github.com/zjc664656505/ LinguaLinked-Inference
gains are more pronounced with larger models, suggesting enhanced scalability and effectiveness in handling complex, resource-intensive tasks. We develop an Android application to demonstrate LinguaLinked’s effectiveness in a typical mobile computing environment, showing how LLMs can operate on devices with diverse capabilities.

heading: 2 Related Works
text: Autoregressive LLMs and Computational Challenges. Recent advancements in NLP have been driven by autoregressive LLMs like GPT-3 (Brown et al., 2020), OPT (Zhang et al., 2022), and LLaMA (Touvron et al., 2023), which generate text sequentially. While effective for tasks such as language generation and translation, their sequential nature leads to computational inefficiencies, especially for longer texts (Lin et al., 2021; Floridi and Chiriatti, 2020; Lee, 2023). Traditionally, these models have been processed on centralized servers (Aminabadi et al., 2022; Borzunov et al., 2022; Du et al., 2023), with innovations aimed at reducing latency and enhancing efficiency (Wang et al., 2023; Romero et al., 2021; Gunasekaran et al., 2022). However, centralization raises privacy concerns and can introduce latency due to data transmission requirements (Khowaja et al., 2023; Sebastian, 2023; Renaud et al., 2023; Kshetri, 2023; Elbamby et al., 2019; Liang et al., 2020a; Park et al., 2019; Mao et al., 2017b). Mobile Constraints and Model Optimization. Deploying LLMs on mobile devices presents challenges due to limited computational and memory resources (Wu et al., 2019; Zhao et al., 2022; Chen and Ran, 2019; Zhang et al., 2019). Techniques like
2
quantization (Gholami et al., 2022; Bondarenko et al., 2021; Coelho et al., 2021), distillation (Liang et al., 2020b; Gu et al., 2023; Jiao et al., 2019), and pruning (Blalock et al., 2020; Hoefler et al., 2021; Liang et al., 2021) help mitigate these issues but may compromise model performance. Frameworks such as TensorFlow Lite (TensorFlow, 2023), TVM (Chen et al., 2018), and ONNXRuntime (Runtime, 2023), along with advanced quantization methods (Yao et al., 2022; Frantar et al., 2022; Xiao et al., 2023), facilitate mobile deployment, yet challenges persist, especially on lower-end devices. Distributed Inference Solutions. Addressing the limitations of single-device deployment, distributed inference strategies like LinguaLinked partition LLMs across multiple devices, reducing the memory load on individual devices and enabling broader device participation in inference tasks. Frameworks such as DeepHome (Hu et al., 2019), MODNN (Mao et al., 2017a), and EdgeFlow (Hu and Li, 2022) have explored data parallelism and model partitioning, primarily for vision models. However, the adaptation of these strategies for LLMs on mobile devices remains an underexplored area, with a need for solutions that consider real-time device performance fluctuations and load balancing (Xu et al., 2022).

heading: 3 LinguaLinked
text: As shown in Figure 2, the LinguaLinked system facilitates LLM distribution across mobile devices by transforming the LLM into a computational graph on a coordinator server, then partitioning it into sub-modules for optimized allocation to devices based on their performance metrics. It employs primary and secondary optimizers for task distribution and load balancing, with a communication strategy that minimizes data transmission between devices, ensuring efficiency and privacy as all data remains local to the devices.

heading: 3.1 System Monitor
text: The system monitor in LinguaLinked is comprised of server and device modules to track and manage performance metrics like bandwidth, latency, memory, and processing speed across devices. The server module controls monitoring activities and processes data for optimization, while the device module assesses performance indicators. Bandwidth is measured by transferring data between devices and calculating the transfer rate, while pro-
cessing speed (FLOP/s) is determined by timing a test model’s execution on each device, offering insights into the system’s operational efficiency.

heading: 3.2 Optimized Model Assignment
text: Subgraph Extraction from LLMs. The first step in preparing LLMs for mobile deployment involves converting them into computational graphs and then segmenting these graphs into smaller, independent subgraphs. These subgraphs are designed to operate separately on different devices, and their extraction is based purely on the computational characteristics of the LLM, without considering the current state of the devices. Nodes within the graph that process inputs from a single node and output to multiple nodes are identified as key points for partitioning. These nodes typically represent distinct layers or operations within the model, making them ideal for creating subgraphs that can be independently executed. The process results in a series of subgraphs, each representing a functional segment of the original LLM, allowing for efficient distribution across the available mobile devices. Subgraph Dependency Search. To handle dependencies between nodes in separate subgraphs of an LLM, we employ a subgraph dependency search algorithm, creating two key maps: the residual dependency map (RDM) and the sequential
3
dependency map (SDM). The SDM tracks direct dependencies between adjacent subgraphs, ensuring that outputs from one subgraph serve as inputs for the next. The RDM identifies dependencies between non-adjacent subgraphs, capturing instances where a subgraph relies on nodes from an earlier subgraph, not directly preceding it. Model Assignment Optimization. After segmenting LLMs into subgraphs, the next step involves assigning these subgraphs as executable sub-modules to mobile devices, considering device constraints and aiming to minimize computation and data transmission times. This involves compiling subgraphs into sub-modules, profiling each for FLOP count, memory needs, and data output size, and then using this information alongside device performance metrics to optimize sub-module allocation. The optimization termed as a primary optimizer, formulated as a linear optimization problem, balances local computation and data transmission efforts to reduce total inference time. Constraints ensure that the memory usage of sub-modules on any device does not exceed a predetermined portion of the device’s available memory.

heading: 3.3 Runtime Load Balancing
text: Load Balancing Optimization. The load balancing mechanism refines the initial model assignment optimization termed as a secondary optimizer by introducing a strategy to overlap sub-modules across devices, categorizing them as movable or unmovable. Movable sub-modules can be dynamically allocated or removed to balance the load, whereas unmovable ones stay fixed. This approach uses linear programming to minimize data transmission and optimize memory usage, enhancing system performance and robustness during intensive tasks. The optimization allows for potential overlaps of sub-modules to the left or right of their current allocation, improving memory utilization within device constraints and facilitating efficient load distribution across the network.

heading: 3.4 Optimized Communication
text: Model Deployment with Load Balancing. LinguaLinked initiates load balancing based on realtime device performance metrics. When an imbalance is detected, it combines the initial model assignment with the secondary optimization to update the task distribution. Unmovable modules remain in place, while movable modules are reassigned as dictated by the load balancing optimization. Devices then adjust their loads according to this new strategy, pausing computations only locally to reduce disruptions. Decentralized Device Communication. In our system, devices communicate in a decentralized, ring-structured manner, where each device sequentially receives, computes, and forwards data to the next device until it reaches the Header again, as illustrated with a solid red line in Figure 4. This efficient communication is facilitated by a message queue utilizing the ROUTER-DEALER pattern, allowing devices to alternate between sending (ROUTER) and receiving (DEALER) roles, which enhances scalability and ensures balanced load distribution. The cycle of data processing involves devices acting first as receivers to perform computations, then as senders to pass on results, maintaining a continuous and organized flow of data throughout the network. Multi-Threaded Inference. Our system implements multi-threaded inference, allowing parallel processing where each thread independently manages a task and progresses to the next one immediately after completion. Due to the nonthreadsafe nature of message queue sockets, we ensure thread safety by using multiple sockets and ports instead of sharing or locking sockets in multi-threads, thereby preventing performance bottlenecks and reducing communication latency. Furthermore, multi-threaded inference boosts CPU efficiency by accommodating varying batch sizes and enabling flexible processing strategies, such as dividing large batches into smaller mini-batches or adjusting batch sizes dynamically, optimizing system performance.
Sequential & Residual Communication. In sequential communication, devices in our system form a circuit where each transmits data to the next in line, creating a flow where only sequential data is exchanged. It leads to inefficiencies as devices pass along residual data not immediately needed by them. To overcome these limitations, we intro-
4
duced a residual communication strategy, allowing for direct transmission of data to target devices, as depicted by green dashed lines in Figure 4. It reduces unnecessary data carriage and latency.

heading: 3.5 Implementation and Methodology
text: LinguaLinked Prototype. We build LinguaLinked atop PyTorch (Paszke et al., 2019), leveraging the torch.fx library (Reed et al., 2022) for computational subgraph extraction and PyTorch sub-module compilation, with performance profiling done via Deepspeed (Rasley et al., 2020). These sub-modules are then prepared for mobile deployment by conversion to ONNX format (Bai et al., 2019) and optimized using int8 precision quantization through ONNXRuntime (Runtime, 2023). Optimization for model distribution and load balancing is achieved with the MILP solver Gurobipy (Gurobi Optimization, LLC, 2023), allowing the deployment of optimized sub-modules on mobile devices through an Android application that utilizes ONNXRuntime’s C++ API. For efficient mobile device communication and distributed inference, ZeroMQ (Zer) with a ROUTERDEALER socket pattern is integrated, enhancing asynchronous communication in the mobile environment. Chat Application. We develop an Android application that allows users to chat with LLMs in a distributed decentralized way as shown in Figure 3. Our application features two distinct modes: header and worker. The header mode focuses on direct user interaction with the LLM such as sending prompts and receiving responses. In contrast, the worker mode dedicates itself to the heavy lifting of model computation, showcasing the synergy between devices to accomplish LLM inference tasks efficiently.

heading: 4 Evaluation

heading: 4.1 Evaluation Setup
text: Hardware. In evaluating the LinguaLinked system, we utilize four mobile devices: three Google Pixel 7 Pros and one CUBOT X30. The specific hardware configurations of these devices are detailed
in Table 1. Our analysis focuses on CPU performance, reflecting the system’s compatibility with the current CPU-only support of ONNXRuntime for LLMs. This approach is deliberate, anticipating future integration with GPU acceleration capabilities as ONNXRuntime evolves, thereby highlighting our system’s adaptability and the consistency of its performance evaluation across varying hardware source. Evaluation Tasks. Our system’s performance is assessed through text generation tasks, using the Wikitext-2 (Merity et al., 2016) with 100 randomly selected samples. Evaluation Models. Evaluation leverages the BLOOM series LLMs (Workshop et al., 2023), including BLOOM 3b, BLOOM 1.7b, and BLOOM 1.1b models, in both full and int8 precision formats. Baseline for Comparison. We established a baseline for assessing on-device distributed inference of LLMs, assigning an equal number of sub-modules (m/n) to each of the n mobile devices, irrespective of their specific hardware or network conditions. This approach allows for a comparison of system throughput between our uniform distribution strategy and both our optimized model assignment and runtime load balancing strategies, in the absence of prior focused research in this area.

heading: 4.2 Performance
text: Optimized Model Assignment Performance. In heterogeneous device environments, optimized model assignment significantly enhances inference throughput compared to baseline strategies, as indicated by the data in Table 2. For int8 quantized models, throughput increases are notable: BLOOM 3b achieves a 1.61× improvement, while BLOOM
5
1.7b and 1.1b models see 1.55× and 1.56× enhancements, respectively. Full-precision models also benefit, with BLOOM 3b, 1.7b, and 1.1b models experiencing improvements of 1.32×, 1.25×, and 1.11×, respectively. The trend is clear—larger models, such as the BLOOM 3b, exhibit greater gains, suggesting that optimization strategies yield more significant benefits for models with higher computational needs. This pattern underscores the efficacy of optimized model assignment in improving the efficiency of model deployment and inference performance in diverse computing landscapes. Multi-threaded Inference Performance. Our study investigates the benefits of multi-threading on inference throughput for both int8 quantized and full-precision BLOOM models, focusing on text generation task as indicated by the data in Table 3. Conducted on three Google Pixel 7 Pro devices, we report that quantized models show a marked performance improvement in multi-threaded setups for text generation. Specifically, the BLOOM 3b quantized model’s throughput increases by 1.81 × with two threads and 2.52 × with five threads compared to a single-threaded baseline. Similarly, the BLOOM 1.7b and 1.1b models demonstrate significant speed-ups, with the 1.7b model doubling its throughput with two threads and reaching a 2.65× increase with five threads, and the 1.1b model achieving a 1.73 × speed-up with two threads and a 2.3× increase with five threads. Full-precision models also benefit from multithreading, albeit to a lesser extent. The BLOOM 3b full-precision model sees a 1.67 × speed-up with two threads and a 1.97 × increase with five threads. The 1.7b and 1.1b models exhibit speed-ups of 1.54 and 1.58 ×, respectively, with two threads, and 1.83 and 1.79 × with five threads.
Micro-Benchmarking the Runtime Load Balancer. In our study, we also investigate the effects of runtime load balancing on the BLOOM 1.7b model in both full precision and int8 quantized formats across three devices, including two high-end and one low-end phone. Initially, model partitions are unevenly distributed, causing the lowend phone to be overloaded. Upon activating the load balancer after processing two samples and continuing with ten more, we observe a significant improvement in processing efficiency. Figure 5 illustrates that enabling load balancing from the second sample noticeably decreases inference latency. For the full precision model, enabling load balancing cuts down latency from 4.624 seconds to 3.587 seconds per token, showcasing an improvement by reallocating 8 sub-modules and overcoming a notable overhead from reloading sessions onto alternate devices. For the quantized model, activating the load balancer reduces latency from 2.756 to 2.087 seconds per token, with a less substantial reloading overhead compared to the full precision model due to the smaller size of quantized sub-modules. Inference times for the quantized model prove to be more stable across generation tasks than for the full precision model. Overall, the implementation of runtime load balancing results in an average improvement of 30% in acceleration, effectively demonstrating the benefits of dynamically adjusting workloads among devices with varying processing capabilities.

heading: 4.3 Sequential and Residual Communication
text: To demonstrate the efficacy of residual over sequential communication, we conducted experiments measuring communication times, employing the Network Temporal Protocol (NTP) for precise syn-
6
chronization and utilizing the system clock for nanosecond accuracy in runtime measurement.
By comparing these methods using three low-
end smartphones and the quantized BLOOM 3b model, recording average delays from ten trials. The results, detailed in Table 4 and Figure 4, show that sequential communication incurs higher delays due to multiple transmission steps and the need for activations to be processed before residual data can be sent. Sequential transmission involved hops with an average data size of 1.5 MB, while residual communication transmitted about 15 KB directly, allowing it to operate in parallel and more efficiently than the sequential method. Note that as the number of residual connections and the size of the model increase, the saved time will likewise increase.

heading: 5 Discussion
text: A major direction for expanding LinguaLinked involves adapting it for distributed fine-tuning on mobile devices, allowing model customization based on user interactions and local data, paving the way for personalized AI applications while preserving data privacy. We also envision extending LinguaLinked to handle multi-modality models, enhancing its applicability in diverse real-world scenarios.
To further improve LinguaLinked, we envision more advanced model computational graph partitioning strategies involving further optimizations on task divisions better aligned with device capabilities. Moreover, integrating advanced load balancing algorithms that account for not only computational capabilities but also battery life and user engagement patterns will ensure a holistic approach to distributed computing on mobile platforms.
Finally, the implications of this research are significant for AI policy, as it challenges the prevailing reliance on cloud infrastructure and centralized data centers for AI deployment. By demonstrating the potential to deploy AI systems from a network of mobile devices, our work suggests a paradigm where the ’means of production’ for AI can be decentralized and localized. This model of deploy-
ment could lead to a future where AI systems are both operated and fine-tuned locally using a diverse array of small devices. Such a setup could make AI systems more difficult to regulate, as the distribution and localization of AI technologies allow for widespread, generic hardware use.

heading: 6 Conclusion
text: In this work, we introduce LinguaLinked, a system for decentralized LLM inference on mobile devices. To the best of our knowledge, LinguaLinked is the first work that exploits deploying LLM distributively on mobile devices. LinguaLinked implemented optimized model assignment strategy, network communication and runtime load balancing mechanism to accelerate the distributed LLM inference on mobile devices. This approach tackles the complexities of deploying both full precision and quantized LLMs of various sizes within mobile computing environments.

heading: 7 Limitations
text: Our results demonstrate promising advancements in distributed LLM inference on mobile devices but also underscore several limitations. Key among these are the overheads from load balancing and constraints of current hardware and software frameworks. As tools like ONNXRuntime evolve to support GPU acceleration, we expect significant enhancements in LinguaLinked’s performance. Furthermore, exploring advanced quantization techniques and communication mechanisms could lead to more efficient distributed inference systems.
At the same time, a critical focus for future iterations of LinguaLinked is energy efficiency. We find that the continuous intensive inference tasks, especially with full-precision models, significantly drain battery life and cause overheating, leading to performance degradation. To address this, we aim to incorporate energy-efficient computing strategies that balance computational demands with energy consumption and thermal management. This could include adaptive algorithms to modulate computational load based on the device’s energy state, and hardware-specific optimizations leveraging low-power processing cores for specific tasks.
Furthermore, when designing our system with privacy in mind, we primarily contrast it with traditional server-based inference systems, operating under the assumption that all inference tasks are conducted locally on ’trusted’ mobile devices. This
7
setup should inherently protect user privacy. However, we recognize potential scenarios where this assumption may fail. For instance, if a device becomes compromised or if unauthorized access is obtained, sensitive local data could be at risk. Moreover, our trust model does not address potential side-channel attacks, which could allow attackers to derive sensitive information from the model’s intermediate activations. These vulnerabilities underscore the need for more comprehensive, multilayered security protocols that extend beyond simple device trust, aiming to robustly safeguard user data in diverse and adversarial environments.
Finally, due to the absence of standardized evaluation benchmarks for distributed LLM inference on mobile devices, we have created our own baselines to assess our system’s performance. However, the lack of universally accepted benchmarks and previous research in this domain complicates the task of conducting thorough comparisons for future work. It is crucial for future research to focus on developing standardized benchmarks in this field. Establishing such benchmarks would facilitate more uniform comparisons between different systems, enhance the clarity of potential improvements, and identify the most effective strategies for distributed LLM inference on mobile platforms.

heading: Future Work
text: At the same time, a critical focus for future iterations of LinguaLinked is energy efficiency. We find that the continuous intensive inference tasks, especially with full-precision models, significantly drain battery life and cause overheating, leading to performance degradation. To address this, we aim to incorporate energy-efficient computing strategies that balance computational demands with energy consumption and thermal management. This could include adaptive algorithms to modulate computational load based on the device’s energy state, and hardware-specific optimizations leveraging low-power processing cores for specific tasks. Furthermore, when designing our system with privacy in mind, we primarily contrast it with traditional server-based inference systems, operating under the assumption that all inference tasks are conducted locally on ’trusted’ mobile devices. This
7
setup should inherently protect user privacy. However, we recognize potential scenarios where this assumption may fail. For instance, if a device becomes compromised or if unauthorized access is obtained, sensitive local data could be at risk. Moreover, our trust model does not address potential side-channel attacks, which could allow attackers to derive sensitive information from the model’s intermediate activations. These vulnerabilities underscore the need for more comprehensive, multilayered security protocols that extend beyond simple device trust, aiming to robustly safeguard user data in diverse and adversarial environments. Finally, due to the absence of standardized evaluation benchmarks for distributed LLM inference on mobile devices, we have created our own baselines to assess our system’s performance. However, the lack of universally accepted benchmarks and previous research in this domain complicates the task of conducting thorough comparisons for future work. It is crucial for future research to focus on developing standardized benchmarks in this field. Establishing such benchmarks would facilitate more uniform comparisons between different systems, enhance the clarity of potential improvements, and identify the most effective strategies for distributed LLM inference on mobile platforms