Empower Large Language Model to Perform Better on Industrial
  Domain-Specific Question Answering
Authors: name: Zezhong Wang; name: Fangkai Yang; name: Pu Zhao; name: Lu Wang; name: Jue Zhang; name: Mohit Garg; name: Qingwei Lin; name: Dongmei Zhang
Published: 2023-05-19 09:23:25

Abstract: Large Language Model (LLM) has gained popularity and achieved remarkable
results in open-domain tasks, but its performance in real industrial
domain-specific scenarios is average since there is no specific knowledge in
it. This issue has attracted widespread attention, but there are few relevant
benchmarks available. In this paper, we provide a benchmark Question Answering
(QA) dataset named MSQA, which is about Microsoft products and IT technical
problems encountered by customers. This dataset contains industry
cloud-specific QA knowledge, which is not available for general LLM, so it is
well suited for evaluating methods aimed at improving domain-specific
capabilities of LLM. In addition, we propose a new model interaction paradigm
that can empower LLM to achieve better performance on domain-specific tasks
where it is not proficient. Extensive experiments demonstrate that the approach
following our model fusion framework outperforms the commonly used LLM with
retrieval methods.