Title: An Approach for Large Language Model Deployment in Distributed Edge Environments
Primary area: foundation or frontier models, including LLMs
Keywords: Large Language Models, Multi-Agent Systems, Edge Devices, Quantization, Question Answering

Abstract: Large Language Models (LLMs) have made a huge impact on tasks like question answering and text generation. However, they still struggle with consistency and efficiency, especially when deployed on resource-limited devices. In this paper, we explore using multiple LLMs to work together in a multi-agent system to improve performance. We use two large language models to generate multiple answers, and then combine the answers using another large language model. We also look at how reducing the precision of these models using quantization affects performance, comparing full-precision and INT8 versions. Our experiments, using questions from the Natural Questions dataset, show that combining smaller models results in more accurate and stable answers than relying on just one model. Quantization helps make the models more efficient, without sacrificing too much performance, making them easier to deploy on devices with limited computational power.

TL;DR: A multi-agent framework combining smaller LLMs with quantization achieves efficient and reliable QA on edge devices.