AllHands: Ask Me Anything on Large-scale Verbatim Feedback via Large
  Language Models
Authors: name: Chaoyun Zhang; name: Zicheng Ma; name: Yuhao Wu; name: Shilin He; name: Si Qin; name: Minghua Ma; name: Xiaoting Qin; name: Yu Kang; name: Yuyi Liang; name: Xiaoyu Gou; name: Yajie Xue; name: Qingwei Lin; name: Saravan Rajmohan; name: Dongmei Zhang; name: Qi Zhang
Published: 2024-03-22 12:13:16

Abstract: Verbatim feedback constitutes a valuable repository of user experiences,
opinions, and requirements essential for software development. Effectively and
efficiently extracting valuable insights from such data poses a challenging
task. This paper introduces Allhands , an innovative analytic framework
designed for large-scale feedback analysis through a natural language
interface, leveraging large language models (LLMs). Allhands adheres to a
conventional feedback analytic workflow, initially conducting classification
and topic modeling on the feedback to convert them into a structurally
augmented format, incorporating LLMs to enhance accuracy, robustness,
generalization, and user-friendliness. Subsequently, an LLM agent is employed
to interpret users' diverse questions in natural language on feedback,
translating them into Python code for execution, and delivering comprehensive
multi-modal responses, including text, code, tables, and images.
  We evaluate Allhands across three diverse feedback datasets. The experiments
demonstrate that Allhands achieves superior efficacy at all stages of analysis,
including classification and topic modeling, eventually providing users with an
``ask me anything'' experience with comprehensive, correct and human-readable
response. To the best of our knowledge, Allhands stands as the first
comprehensive feedback analysis framework that supports diverse and customized
requirements for insight extraction through a natural language interface.