Octopus v2: On-device language model for super agent
Authors: name: Wei Chen; name: Zhiyuan Li
Published: 2024-04-02 09:01:32

Abstract: Language models have shown effectiveness in a variety of software
applications, particularly in tasks related to automatic workflow. These models
possess the crucial ability to call functions, which is essential in creating
AI agents. Despite the high performance of large-scale language models in cloud
environments, they are often associated with concerns over privacy and cost.
Current on-device models for function calling face issues with latency and
accuracy. Our research presents a new method that empowers an on-device model
with 2 billion parameters to surpass the performance of GPT-4 in both accuracy
and latency, and decrease the context length by 95\%. When compared to Llama-7B
with a RAG-based function calling mechanism, our method enhances latency by
35-fold. This method reduces the latency to levels deemed suitable for
deployment across a variety of edge devices in production environments,
aligning with the performance requisites for real-world applications.