Kati8/ICLR_2023_2025 huggingface dataset:
authors: Julian Hao Yong
authors: Mei Kuan Lim
authors: Chun Yong Chong
reviews: 
Comments: With the increase in number of patients requiring mental health care, the cognitive load on mental health professionals has increased a lot. A major part of it is referring to notes from the past sessions. Due to visual information being easier to process and remember, the authors propose a visual diagram based approach to summarize past notes instead of just pure text based summaries. They use a simple LLM based prompting mechanism to take in the transcript and output the summary in a structured way that is parsed into a PlantUML mind map syntax. They also build a web tool that can be used for this.
Strengths: - The authors provide a web based tool that can be used by mental health professionals for building mind-map visual diagrams of past notes
- They show that their tool can build reasonably good mind maps evaluated using human experts.
Weaknesses: - The paper lacks novelty - as it is merely an application of using LLMs for a very specific use case. The prompting itself is also not smart or novel in any way as it just has a structured output that is parsed into a mind-map. I dont believe this would be of interest to many people.
- There are no comparisons to text based summaries for this use case and it is not clear how these visual summaries could be more useful to a mental health professional. Several ablations are missing.
- The contributions is really just the prompt and the web based tool.
- There is also no comparison between the use of different kind of models / LLMs use
Recommendation: 1
Reviewer confidence: 5
Soundness/correctness: 2
Clarity: 3
reviews: 
Comments: The authors propose a method to construct a summarised mind map of counselling transcripts. They use the MEMO dataset, from which the authors extract 20 samples and perform their analysis on them. They developed a web-based tool to showcase the generated mind maps.

While the topic of visual summaries is relevant to explore, this paper tackles it from an engineering perspective rather than a research perspective. This might make a good demo paper in some conference, however, I am not so sure about this being a part of the research main track.
Strengths: - Interesting problem of visual summaries is explored. Mind maps are an effective and quicker way of information communication, so using such things to help therapist is a good idea.
- It is good that the authors have developed a web based tool which can help people.
- The paper is easy to follow.
Weaknesses: - No substantial contribution is there in this paper for it to be a part of main research track, maybe a better fit for demo track for some conference.
- Only 20 mind maps evaluated by only 3 people with using just 1 LLM as the summariser might not be enough to be considered as a thorough investigation. I would suggest the authors to conduct a more detailed analyses by using more LLMs in the place of GPT-4o Mini to summarise the transcripts and then perform a comparison of different LLMs here.
- In a nutshell, the paper does not propose a new problem, does not have a new dataset, does not propose any novel method, and does not contain any interesting observations or analysis. It is a very engineering pov paper, which is not a good fit for this venue.
Recommendation: 1
Reviewer confidence: 4
Soundness/correctness: 2
Clarity: 2
reviews: 
Comments: The paper introduces a web-based summarization tool that uses Large Language Models (LLMs), specifically GPT-4o Mini, to transform counseling session transcripts into visual mind map summaries. By leveraging the MEMO dataset of counseling conversations, the tool extracts key points from qualitative dialogues and organizes them hierarchically using PlantUML syntax. Human evaluations based on accuracy, completeness, conciseness, and coherence indicate that the generated mind maps effectively capture essential information. The tool demonstrates potential in aiding therapists to quickly recall session details and reduce cognitive load by providing a clear, visual overview of counseling sessions.
Strengths: The integration of LLMs, specifically GPT-4o Mini, with PlantUML for generating mind maps from qualitative counseling data is a creative and original contribution. This approach moves beyond traditional text-based summaries by providing a visual representation of complex counseling sessions, which is underexplored in the mental health domain.
Weaknesses: 1. Limited Sample Size and Representativeness

The study utilizes only 20 randomly selected samples from the MEMO dataset for the evaluation. This small sample size is not sufficiently representative of the diverse range of counseling conversations that occur in real-world settings. As a result, the findings may not generalize well to broader applications. To strengthen the validity of the results, it would be beneficial to include a larger and more varied dataset. This expansion would help in capturing a wider array of counseling scenarios and linguistic nuances, thereby enhancing the robustness of the tool's performance and its applicability to different contexts.

2. Reliance Solely on Human Evaluation Lacking Reproducibility

The evaluation of the generated visual summaries is based entirely on human assessments from a small group of participants who are researchers in the field of Information Technology, not mental health professionals. This approach raises concerns about the reproducibility and objectivity of the results. Human evaluations can be subjective and may vary significantly between different evaluators. Incorporating quantitative evaluation metrics, such as adapted versions of ROUGE or BLEU scores for summarization tasks, could provide more objective measures of the tool's performance. Additionally, involving mental health professionals in the evaluation process would offer insights that are more aligned with practical therapeutic needs, thereby increasing the reliability and validity of the findings.
Recommendation: 3
Reviewer confidence: 4
Soundness/correctness: 1
Clarity: 3