DialogStudio: Towards Richest and Most Diverse Unified Dataset
  Collection for Conversational AI
Authors: name: Jianguo Zhang; name: Kun Qian; name: Zhiwei Liu; name: Shelby Heinecke; name: Rui Meng; name: Ye Liu; name: Zhou Yu; name: Huan Wang; name: Silvio Savarese; name: Caiming Xiong
Published: 2023-07-19 17:57:53

Abstract: Despite advancements in conversational AI, language models encounter
challenges to handle diverse conversational tasks, and existing dialogue
dataset collections often lack diversity and comprehensiveness. To tackle these
issues, we introduce DialogStudio: the largest and most diverse collection of
dialogue datasets, unified under a consistent format while preserving their
original information. Our collection encompasses data from open-domain
dialogues, task-oriented dialogues, natural language understanding,
conversational recommendation, dialogue summarization, and knowledge-grounded
dialogues, making it an incredibly rich and diverse resource for dialogue
research and model training. To further enhance the utility of DialogStudio, we
identify the licenses for each dataset and design domain-aware prompts for
selected dialogues to facilitate instruction-aware fine-tuning. Furthermore, we
develop conversational AI models using the dataset collection, and our
experiments in both zero-shot and few-shot learning scenarios demonstrate the
superiority of DialogStudio. To improve transparency and support dataset and
task-based research, as well as language model pre-training, all datasets,
licenses, codes, and models associated with DialogStudio are made publicly
accessible at https://github.com/salesforce/DialogStudio