ProcessChat: A Business Process Grounded Dialogue Dataset
Authors: Neelamadhav Gantayat, Avirup Saha, Renuka Sindhgatta


Reviewer paper summaries: The paper proposes a dataset of conversations grounded on business process description and model specification, called ProcessChat. The conversations in the dataset are grounded on 73 Business Process Model and Notation processes and corresponding natural language descriptions. Specifically, the dialogues are constructed first by creating dialogues from natural language descriptions, after which LLM is called to generate the appropriate assistant response. The authors also provide a baseline system based on prompt-tuning and finetuning-based LLMs.
Summary of strengths: The paper addresses a previously overlooked area of natural language application of business processes, and contributes a valuable and well-constructed resource to the field.

The authors tried many different models and consider multiple types of text similarity metrics.
Summary of weaknesses: While the authors did try many different models, the lessons from comparing different types of finetuning and prompting-based models are unclear to me. 

The evaluation only focuses on one task (assistant response generation), and also lacks human evaluation. Since the dialogues concern a specialized field, ideally an expert could have been consulted for evaluation or at least a qualitative assessment of the framework. The most ideal case would be a real deployment and user study, but I understand that this can be difficult.
Comments, suggestions and typos: The figures of this current version is not very legible. I believe the authors can find a way to make the fonts larger while having the same information (esp Figures 1, 2, 3). Also, colors can be used in lieu with flow diagram shapes.

Organization of contents can be improved. There are occasionally long paragraphs with a lot of information (ex: Process Description in CNL). Organizational tools like headers, bulletpoints can be used to break up these paragraphs and emphasize the focus of the content.

Including sample responses of different model given a dialogue history and flow in the appendix can help the readers better understand the differences of the models. This could be done not necessarily for all models, but for each class (e.g. LoRA, 5-shot, 4-shot prompts, etc)

Figure 1 does not represent the paper well. It is just an example BPMN process. A better figure would be ProcessChat dialogue paired with this process, since the dialogues are what this work is about.

Overall, business process is a relatively less known domain. I suggest the authors augment its introduction and potential value in the paper (i.e. in Figure 1, explain how it can be useful in the introduction,.)

For missing human evaluation, one (incomplete) replacement could be to use LLM, and include the used LLM evaluation prompt in the appendix.
Overall assessment scores: 2.5