Orchestrating Heterogeneous Architecture for Fast Inference of Mixture-of-Experts Models
Authors: Keisuke Kamahori, Tian Tang, Yile Gu, Kan Zhu, Baris Kasikci


Reviewer paper summaries: This paper introduces a latency model for Mixture of Experts (MoE) Large Language Models (LLMs) aimed at optimizing the execution strategy for MoE layers between CPU and GPU devices under conditions of limited GPU memory. The proposed approach, named Twiddler, initially loads all non-MoE layers into the GPU memory and allocates MoE experts to the CPU memory. Twiddler then dynamically transfers frequently accessed experts to the GPU, maximizing their availability within the constraints of GPU memory.

During inference, Twiddler prioritizes executing most tokens on the GPU, transferring experts from CPU to GPU as necessary, based on a predefined threshold that dictates when an expert's involvement is critical enough to warrant this transfer. This strategy minimizes CPU usage to only those computations for tokens that fall below the threshold, thereby optimizing performance.
Finally, the Twiddler performs better in various applications than in existing systems.
Summary of strengths: 1. Efficiency and simplicity in implementation.
2. Maintained performance advantage in beam search scenarios.
Summary of weaknesses: 1. The experimental setup utilized a single GPU with a high-performance CPU, which may not reflect typical configurations where a powerful GPU is paired with a less capable CPU. Testing with a more suitable CPU (i.e., weaker CPUs) could better demonstrate Twiddler’s robustness.
2. The evaluation focused on a single dataset, which might not adequately represent the different routing behaviors encountered across different datasets. A broader evaluation is necessary to validate the method's general effectiveness.
3. The end-to-end performance was limited to maximum input lengths of less than 1024 tokens. Exploring performance with longer inputs, such as lengths of 1024, 1536, and 2048 tokens, is crucial to understanding the method's scalability.
4. The experiments were confined to the Mixtral-8x7B model. Extending the application of Twiddler to more MoE LLMs (e.g. Deepseek-MoE, switch-transformer, and LLaMA-M
Comments, suggestions and typos: Please see above
Overall assessment scores: 2.5