AutoRAG-HP: Automatic Online Hyper-Parameter Tuning for
  Retrieval-Augmented Generation
Authors: name: Jia Fu; name: Xiaoting Qin; name: Fangkai Yang; name: Lu Wang; name: Jue Zhang; name: Qingwei Lin; name: Yubo Chen; name: Dongmei Zhang; name: Saravan Rajmohan; name: Qi Zhang
Published: 2024-06-27 15:18:21

Abstract: Recent advancements in Large Language Models have transformed ML/AI
development, necessitating a reevaluation of AutoML principles for the
Retrieval-Augmented Generation (RAG) systems. To address the challenges of
hyper-parameter optimization and online adaptation in RAG, we propose the
AutoRAG-HP framework, which formulates the hyper-parameter tuning as an online
multi-armed bandit (MAB) problem and introduces a novel two-level Hierarchical
MAB (Hier-MAB) method for efficient exploration of large search spaces. We
conduct extensive experiments on tuning hyper-parameters, such as top-k
retrieved documents, prompt compression ratio, and embedding methods, using the
ALCE-ASQA and Natural Questions datasets. Our evaluation from jointly
optimization all three hyper-parameters demonstrate that MAB-based online
learning methods can achieve Recall@5 approx 0.8 for scenarios with
prominent gradients in search space, using only sim20% of the LLM API calls
required by the Grid Search approach. Additionally, the proposed Hier-MAB
approach outperforms other baselines in more challenging optimization
scenarios. The code will be made available at https://aka.ms/autorag.