Venue: NAACL 2025

Afrispeech-Dialog: A Benchmark Dataset for Spontaneous English Conversations in Healthcare and Beyond

Mardhiyah Sanni\({}^{1,2}\), Tassallah Abdullai\({}^{2,3}\), Devendra D. Kayande\({}^{1,2,4}\), Emmanuel Ayodele\({}^{1}\), Naome A. Etori\({}^{2,5}\), Michael S. Mollen\({}^{2,6}\), Moshood Yekini\({}^{2}\), Chibuzor Okocha\({}^{2,7}\), Lukman E. Ismaila\({}^{2,8}\), Folafunnil Omofoye\({}^{2,9}\), Boluwatife A. Adewale\({}^{2}\), Tobii Olatunji\({}^{1,2,10\lx@paragraph}\)

###### Abstract

Speech technologies are transforming interactions across various sectors, from healthcare to call centers and robots, yet their performance on African-accented conversations remains underexplored. We introduce Afrispeech-Dialog, a benchmark dataset of 50 simulated medical and non-medical African-accented English conversations, designed to evaluate automatic speech recognition (ASR) and related technologies. We assess state-of-the-art (SOTA) speaker diarization and ASR systems on long-form, accented speech, comparing their performance with native access and discover a 10%+ performance degradation. Additionally, we explore medical conversation summarization capabilities of large language models (LLMs) to demonstrate the impact of ASR errors on downstream medical summaries, providing insights into the challenges and opportunities for speech technologies in the Global South. Our work highlights the need for more inclusive datasets to advance conversational AI in low-resource settings.

1
Footnote 1: Authors marked with \(\bullet\) are senior authors.

_Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 8399–8417

April 29 - May 4, 2025 \(\copyright\)2025 Association for Computational Linguistics

## 1 Introduction

Deep Learning approaches have revolutionized and yielded significant performance gains across several natural language tasks Bharadiya (2023), especially for high-resource languages Radford et al. (2023). While conversational speech recognition has continued to make significant strides in task automation in different domains such as medical Biswas and Talukdar (2024); Abacha et al. (2023); Yim et al. (2023), voice assistants Pasandi and Pasandi (2022); Mani et al. (2020), call center Plaza et al. (2021), and robotics Skantze (2021), much of the research in Automatic Speech Recognition (ASR) has focused on monolingual speech with native accents Aksenova et al. (2022), with considerable performance gaps in diverse linguistic, low resource, and accented contexts Radford et al. (2023); Olatunji et al. (2023).

In Anglophone African contexts, regional accents further complicate the development of ASR systems resulting in poor generalization from state-of-the-art (SOTA) models Hinsvark et al. (2021) in both general Adelani et al. (2022); Olatunji et al. (2023); Ogun et al. (2024); Tonja et al. (2024) and medical Afonja et al. (2024); Olatunji et al. (2023) contexts. In more developed countries, Performat Speech recognition models are particularly useful in overburdened healthcare systems, where they can help to reduce the documentation workload for overwhelmed clinicians Afonja et al. (2024); Olatunji et al. (2023).

Recent advancements in medical conversation transcription and summarization using LLMs have led to wide adoption of these technologies in hospitals in developed countries Michalopoulos et al. (2022); Yim et al. (2023) helping to improve clinical documentation quality and productivity Galloway et al. (2024). Given the heavy patient loads Baye et al. (2020); Etori et al. (2023) as well as the recent integration of AI/NLP/ASR systems into clinical workflows in African clinics 12, performant ambient transcription and summarization systems are highly desired. However, the generalizability of such systems to African contexts remains underexplored Afonja et al. (2024).

Footnote 1: [https://techcabal.com/2023/05/29/intron-health-brings-ai-to-aftrican-healthcare/](https://techcabal.com/2023/05/29/intron-health-brings-ai-to-aftrican-healthcare/)

To tackle this problem, our contributions are as 

follows:

1. AfriSpeech-Dialog a novel 7hr dataset of 50 long-form simulated African-accented English medical and non-medical conversations
2. Benchmark the performance of SOTA speaker diarization, speech recognition, and LLM-based summarization on African-accented conversations

We lay the groundwork for more inclusive ASR and NLP technologies in healthcare and beyond.

## 2 Related Work

### ASR in Medical Conversations and Summarization

The role of ASR in medical documentation has grown significantly, particularly in telehealth and in-person patient-physician consultations Korfiatis et al. (2022); Galloway et al. (2024); Michalopoulos et al. (2022); Yim et al. (2023). Accurate ASR in medical dialogue is critical, as transcription errors can lead to incorrect medical records. Several datasets have been developed to facilitate the study of ASR in medical contexts. PriMock57 Korfiatis et al. (2022) provides primary care mock consultations in European English accents; Fareez et al. (2022) evaluates medical ASR on simulated patient-physician medical interviews with a focus on respiratory cases; Enarvi et al. (2020) experiments automatically generated medical reports from diarized doctor-patient surgery conversations. Le-Duc et al. (2024) proposed a real-time speech summarization system for medical conversations with the ability to generate summaries for every (local) and end of (global) utterances, eliminating the need for continuous update and revision of the summary state.

These datasets primarily focus on non-African accents and therefore do not account for the challenges specific to African-accented medical speech.

### Non-medical conversational ASR

Pezik et al. (2022) released DiaBiz, an Annotated Corpus of over 400hrs of Polish call center dialogs. Other conversational, parliamentary, oratory datasets like AMI Carletta et al. (2005), Earnings22 Del Rio et al. (2022), Voxpopuli Wang et al. (2021) have gained popularity on public ASR benchmarks. Conversational ASR has also been explored in other domains such as call centers Plaza et al. (2021), and robotics Skantze (2021). However, these datasets lack representation of African-accented speech.

### African Accented ASR

There has been growing interest in developing ASR systems that cater to African languages; for example, Yilmaz et al. (2018) developed a multilingual ASR system for code-switched South African speech. Multilingual ASR systems, such as EVI dataset Spithourakis et al. (2022), offer a strong foundation for developing similar models in African contexts where data scarcity hinders progress. Olatunji et al. (2023b) released a pan-African accepted English dataset for medical and general ASR. While these datasets focus on single-speaker speech recognition, AfriSpeech-Dialog is the first African-accented English conversational dataset spanning medical and non-medical domains, enabling additional tasks like diarization and summarization.

### Speaker Diarization in Multi-Speaker Conversations

To increase the efficiency of NLP/ASR systems, enormous contributions were made to researching the integration of speaker diarization (SD) into its pipeline. Serikov et al. (2024) provides a comparative analysis of SD models - Pyannote Bredin et al. (2020), CLEVER3, and NVIDIA NeMo Harper et al. (2019) on 20 different German dialects for diarization and identification (DI) task. NVIDIA NeMo performs slightly better with a competitive performance due to its multiscale segmentation for identifying and removing shorter segments. On a similar DI task, Chua et al. (2023) also benchmarked the performance of multilingual ASR models in open and closed tracks on the challenging MERLIon CCS English - Mandarin datasets - an extracted spontaneous and codeswitching parent-child conversation speeches. However, SD for African-accented conversations remains underexplored. Benchmarking SOTA SD models on AfriSpeech-Dialog reveals their limitations in this setting.

Footnote 3: [https://www.oxfordwaveresearch.com/products/cleaver/](https://www.oxfordwaveresearch.com/products/cleaver/)

## 3 Methodology

Figure 1 shows an overview of the dataset creation and benchmarking process, illustrating how AfriSpeech-Dialog supports tasks like speaker diarization, ASR, and summarization. Below, we



describe the dataset creation process and the evaluation of state-of-the-art (SOTA) models on these tasks to demonstrate their applicability and highlight challenges in African-accented conversational speech.

### Dataset Statistics

Here, we outline our dataset creation process.

#### 3.1.1 Collecting Conversations

We recorded simulated virtual and in-person medical and non-medical conversations from African medical and non-medical crowdworkers on the Intron Platform 4 similar to the process described in Olatunji et al. (2023b). Each conversation began with speakers providing consent, and any identifiable information in the consent segment was removed.

Footnote 4: [https://speech.intron.health](https://speech.intron.health)

For medical conversations, following the process described in Fareez et al. (2022) and Korfiatis et al. (2022), clinical experts prepared "patient cards" with African-context disease conditions and demographics. Doctor and Patient actors included medical professionals (e.g. doctors, nurses) familiar with Objective Structured Clinical Examinations (OSCE), a widely used assessment in medical education that simulates doctor-patient interactions (Fareze et al., 2022). Each patient actor was provided with a detailed "patient card" that included information on their condition, demographics, and medical history, as shown in Table 2. Consistent with OSCE format, patient cards were hidden from doctor actors to facilitate a more natural consultation.

For general domain conversations, participants engaged in open discussions based on "topic cards" prepared by a team of reviewers. Each card contained a conversation topic, a brief description, and two discussion prompts to guide the conversation. The pair of participants (actors) had prior access to the cards and were advised to read through and understand them before starting the conversation. Table 3 shows a sample topic card.

The conversation recordings were stored as mono-channel, 16-bit wav files, with a 48 kHz sampling rate. A team of clinician reviewers reviewed the conversations and selected a high quality subset for this release. The dataset was collected across three African countries--Nigeria, Kenya, and South Africa. The speakers represent a diverse range of accents (11 in total): Hausa, Isoko, Idoma, Urhobo, Ijaw, Yoruba, Swahili, Sesotho, Igbo, Igala, and Ebira.

#### 3.1.2 Recording Characteristics

Dataset statistics are summarized in Table 1. The dataset features two speakers in each conversation

\begin{table}
\begin{tabular}{l c c} \hline \hline  & **Medical** & **General** \\ \hline
**Counts** & 20 & 29 \\
**Timestamp Counts** & 9 & 21 \\
**Avg. Num. of Turns** & 78.6 & 30.55 \\
**Total Duration (hours)** & 2.07 & 4.93 \\
**Avg. Word Count** & 725.3 & 1356.83 \\
**Num. of Countries** & 1 & 3 \\
**Num. of Accents** & 6 & 8 \\
**Gender (M,F)** & (14, 26) & (25, 33) \\ \hline \hline \end{tabular}
\end{table}
Table 1: _Statistics of the **medical** and **non-medical** datasets.

Figure 1: AfriSpeech Dialog: Dataset and Benchmarking Pipeline



in both the medical and general domains. Medical conversations were more structured, with doctors asking direct questions and patients responding, resulting in a more formal exchange. General conversations were more relaxed, with spontaneous discussions on various topics. Overlapping speech occurs occasionally but is usually brief, involving short interjections like "yes" or "okay." Disfluencies and some code-switching reflect the natural flow of African English speakers. General conversations have fewer speaker turns but a higher average word count compared to medical ones, as speakers tend to talk longer to express their thoughts. These characteristics make the dataset valuable for testing speaker diarization and transcription models on African-accented speech.

#### Transcription Process

All conversations were manually transcribed by five professional annotators selected from top-performing contributors on the Intron Platform. Annotators were instructed to annotate speaker turns and insert timestamps for each turn, with all annotators required to be familiar with medical terminology. To ensure quality, clinician reviewers evaluated a random 20% of each annotator's work for accuracy, with at least 90% correctness threshold required for inclusion. Contributors and annotators were paid $10-$20 per hour depending on task complexity and clinical experience. The dataset is released under a CC-BY-NC-SA 4.0 License.

\begin{table}
\begin{tabular}{l l} \hline
**Condition** & **Malaria** \\ \hline
**Demographic (Age, Gender)** & 32-year-old Female \\
**Presenting Complaint** & Fever and chills (2 days) \\
* & \\
* Fever for 2 days (High grade, not relieved by medication) \\
* Chills for 2 days (Intermittent, severe) \\
* Headache for 2 days (Generalized, throbbing, 7/10 in severity) \\
* Fatigue and general body weakness \\
* No cough, diarrhea, vomiting, or urinary symptoms \\
* Patient lives in malaria-endemic area; no recent travel history \\
**Past Medical History (PMH)** & No chronic disease or surgery \\
**Family History** & No family history of similar illness \\
**Social History** & Does not drink alcohol or smoke \\
**Allergy History** & No known allergies \\ \hline \end{tabular}
\end{table}
Table 2: Example Patient Card for Medical Conversations.

\begin{table}
\begin{tabular}{p{85.4pt} p{267.8pt}} \hline
**Topic: Cyberbullying** \\ \hline
* Cyberbullying is a form of bullying that takes place online or through electronic communication. It involves using technology (social media, text messages, online forums) to intimidate or humiliate someone. Examples include:
* Spreading rumors
* Sending hurtful messages
* Sharing embarrassing information without consent \\

* What steps do you take to protect yourself from cyberbullying?
* Do you think social media has effective policies in place, or could they improve? \\ \hline \end{tabular}
\end{table}
Table 3: Sample Conversation Card for General Conversations.
 



### Speaker Diarization

To benchmark diarization performance on this dataset, we selected three recent high-performing neural diarization models:

* **Pyannote**[14]: This model leverages a pre-trained neural network that computes x-vector embeddings for speaker diarization. It uses an ECAPA-TDNN (Desplanques et al., 2020) architecture for speaker embeddings, shown to improve speaker separation in diarization tasks.
* **Reverb diarization v2**[3]: This model is an extension of Pyannote 3.0, fine-tuned on 26,000 hours of annotated speech. However, this model uses WavLM (Chen et al., 2022) instead of the SincNet (Ravanelli and Bengio, 2018) features in the Pyannote 3.0 basic model.
* **Titanet**: This Nvidia diarization pipeline uses MarbleNet (Nguyen-Vo et al., 2022) for voice activity detection and Titanet-L (Koluguri et al., 2022) for embedding extraction. Titanet uses a 1D depth-wise separable convolutions with Squeeze-and-Excitation (SE) layers with global context, followed by a channel attention-based statistics pooling layer to map variable-length utterances to fixed-length embeddings (t-vectors) (Koluguri et al., 2022).

#### 3.2.1 Evaluation Metrics

We evaluated diarization performance using the **Diarization Error Rate (DER)**[5]. DER quantifies the percentage of time that speakers are misattributed or missed in the diarization output.

We used the _pyannote.metrics_ library 5 to calculate DER for each recording and computed the absolute DER for the entire dataset. The optimal mapping between the reference and hypothesis labels was obtained using the Hungarian algorithm (Kuhn, 1955), ensuring an accurate alignment. We report DER on medical and general domain conversations.

Footnote 5: [https://pyannote.github.io/pyannote-metrics/](https://pyannote.github.io/pyannote-metrics/)

### Automatic Speech Recognition (ASR)

To benchmark ASR performance, we compare SOTA open source pretrained models, Whisper (Radford et al., 2023), Distil-Whisper (Gandhi et al., 2023), Nvidia Parakeet (Harper et al., 2019), Canary (Puvvada et al., 2024), MMS (Pratap et al., 2024) and Wav2vec2 (Baevski et al., 2020).

Open-source models offers transparency and reproducibility. They are often trained on diverse, real-world speech data, provide competitive baselines for dialog-specific tasks such as handling speaker variation, spontaneous speech, and overlapping speakers.

#### 3.3.1 Preprocessing

The original transcripts contained timestamps and speaker tags. We removed these items from the text as they are unnecessary for the ASR task. Long form audio recordings exceeded the context length of most ASR models. They were therefore chunked into 30 second segments for inference and transcript segments returned were concatenated.

#### 3.3.2 Evaluation Metrics

ASR performance was evaluated using **Word Error Rate (WER)**. WER measures the total number of insertions, deletions, and substitutions in the predicted text with respect to the total number of words in the reference text.

### Medical Conversation Summarization

Nine LLMs (large, small, open, closed, general, and biomedical) were benchmarked for summarizing doctor-patient dialogues. Each LLMs was presented a human conversation transcript and was prompted (Appendix section D) to generate a detailed summary.

Closed-source general LLMs:OpenAI (GPP-4o, GPT-3.5-turbo) (Achiam et al., 2023), and Anthropic Claude-3-Sonnet(Anthropic, 2023) represent leading general-purpose commercial LLMs.

Open-source small general LLMs:Meta-Llama-3.1-8B-Instruct, Meta-Llama-3.2-3B-Instruct (Dubey et al., 2024), Microsoft-Philim-3-mini-4k-instruct (Abdin et al., 2024), and Google-Gamma-2-9b-it (Team et al., 2024) selected for their instruction-following abilities or multilingual support, which is essential for code-switching.

Biomedical open-source LLMs:m42-health-Llama3-Med42-8B(Christophe et al., 2024), johnsnowlabs-JSL-MedLLama-3-8B-v2.0, selected for their biomedical adaptation. Examples of generated summaries are provided in Appendix



#### 3.4.1 Quantitative Evaluation

We used the BERTScore Zhang et al. (2019) to evaluate the quality of the LLM-generated summaries against the expert-generated reference summaries. Although BERTScore is widely used, studies like Hanna and Bojar (2021) have shown its limitations, particularly in capturing fine-grained semantic nuances and penalizing stylistic differences.

#### 3.4.2 Qualitative Evaluation

To address the limitations of BERTScore, we complement it with two qualitative evaluation approaches Human and LLM-as-Judge where summaries were evaluated on a 5-point scale adapted from Zheng et al. (2023); Liu et al. (2023), 1 (worst) to 5 (best) on the following six criteria: recall of the diagnosis, accuracy of the treatment plan, avoidance of false or fabricated information, clarity and structure, and inclusion of important positive and negative clinical details. If any criterion (e.g., treatment plan) was absent from the conversation (transcript), a score of 0 was to be assigned for that criterion. Detailed evaluation criteria are in the Appendix Section A.

LLM-as-JudgeConsistent with the growing trend in recent studies Zheng et al. (2023); Liu et al. (2023) we used generative models for automated summary evaluation. We used the OpenAI's "o1" model Temsah et al. (2024) prompting based on the criteria mentioned above. Detailed verbatim prompts and the method for computing accuracy scores are provided in Appendix Section B and C.

Human EvaluationIn a blind study, we randomly present pairs of human transcripts and LLM-or human-generated summaries to a team of 4 clinical experts. The experts compared the information available from summaries to conversation transcripts using the 6 criteria listed above Kaniithi et al. (2024); Singhal et al. (2023); Wang et al. (2023). Each summary was independently rated by 2 experts.

#### 3.4.3 Error Propagation on Cascading Models

Since real-world conversation summarization systems rely on imperfect ASR transcripts and accented medical ASR transcription is challenging for several ASR systems Afonja et al. (2024), we further evaluated summaries generated based on predicted (machine) transcripts to determine if there was a drop in quality when compared with summaries generated based on human transcripts Giuseppe et al. (2021). We measured summary quality using LLM-as-Judge.

## 4 Experiments

### Diarization

We download and run inference using publicly released checkpoints from Hugging Face Wolf et al. (2020), with default hyperparameters. We set the collar to 0.0, meaning no margin was allowed around speaker transitions, ensuring that even short overlaps (e.g., "yes" or "okay") were evaluated directly without any tolerance. Overlapping speech was also not excluded from the evaluation.

We ran inference on a single Nvidia T4 GPU. Inference for Pyannote and Reverb took approximately 2 hours while the Titanet took about 30 minutes. Results represent single runs.

### Asr

Models were downloaded from publicly available huggingface Wolf et al. (2020) checkpoints with default hyperparameters and the default generation configuration was used. We ran inference on Nvidia T4 GPUs. The inference required an average of around 30 minutes for the whole dataset for the open-source models. Results represent single runs.

### Summarization

For open-source LLMs, we used publicly available checkpoints from HuggingFace Wolf et al. (2020) without altering their default hyperparameters, except for setting max_new_tokens to 1024. Closed-source models were accessed via their respective APIs, also using default hyperparameters. The prompt template was adapted from prior work Zheng et al. (2023); Liu et al. (2023), and to ensure consistency, the same prompt was used across all models (details can be found in the Appendix).

We conducted the summarization experiments under two scenarios: (1) generating summaries from human-produced transcripts and (2) generating summaries from transcripts created by our best-performing ASR model (Whisper-large-v3).

## 5 Results and Discussion

### Diarization

We compute DER separately for a subset of conversations with accurate timestamps- medical (9

samples, Med DER) and general (21 samples, Gen DER) domain conversations. The results are shown in Table 4 and Figure 2. We also show the performance of these models on conversational datasets on other accents, using the values reported in [37; 50].

The models consistently performed better on general domain conversations compared to medical conversations, likely due to their relaxed structure and fewer interruptions. Results show diarization results on Afrispeech-Dialog are better than AMI and DIHARD likely because of the simulated and structured nature of conversations.

### Asr

We report WER for all conversations, as well as separately for the medical and non-medical portions of the data (Table 5). The **openai/whisperlarge** models achieved the best performance, followed by **nvidia/canary**, **nvidia/parakeet** and models and lastly **wav2vec2** based models.

The results, as presented in Table 5 and Appendix Figure 3, demonstrate a clear trend: models exhibit superior performance on non-medical (general domain) audio compared to medical domain audio, roughly 5% better WER, suggesting medical conversations are more challenging likely due to accepted medical jargon such as medication names, diagnoses, and density of numerical entities [6].

Following trends in Olatunji et al. (2023b), wav2vec2-large-960h finetuned exclusively on read librispeech audiobooks [8] performs worst, confirming that model size (number of parameters), training set size, training data domain (in-the-wild, conversational) and multilingual pretraining play a role in generalizability to accented speech.

Additionally, we compare performance on other conversational datasets reported in Table 5 and show a 5 to 20 point (absolute) performance drop on average suggesting poor generalizability of SOTA models to African-accented conversation English speech. This degradation underscores the challenges ASR models face with African accents, highlighting the pressing need for high-quality, domain-specific datasets to improve ASR performance for low-resource accents and languages.

### Summarization

Table 6 shows our summarization results.

In BERTScore-H, summaries are generated based on human transcripts and evaluated against human reference summaries. In BERTScore-M, summaries are generated based on machine-predicted transcripts (Whisper-Large-v3) and evaluated against human reference summaries.

LLM-Eval-H represents the LLM-as-Judge results when key details in the human transcripts are sought from summaries generated from human transcripts. LLM-Eval-M represents the LLM-as-Judge results when key details in the human transcripts are sought from summaries generated from archive-predicted transcripts.

Table 6 shows that general domain models like GTP4o and Claude consistently achieve high scores across all evaluation metrics followed by Biomedical LLMs where generated summaries are semantically more similar to reference summaries.

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline
* \\ Pyannote & 21.30 & **31.46** & 19.09 & 24.8 (4.6) & 34.4 \\ Reverb & 26.87 & 58.04 & 20.10 &
- \\ \hline \hline \end{tabular}
\end{table}
Table 4: _Diarization Error Rate (DER) for all 30 audios, with detailed results for the **Medical (Med. DER)** and **General (Gen. DER)** subsets. The **AMI DER** and **DIHARD DER** columns show performance on the **AMI MixHeadset** [37] and **DIHARD II** [38] datasets, respectively. **Lower DER is better**, and (*) indicates results where overlapping speech regions were ignored._

Figure 2: Comparison of Medical and General DER for Different Models



Claude-3 was ranked highest for summarizing human transcripts, while Gemma-2 ranked highest for summarizing ASR transcripts.

Open general smaller LLMs like Gamma and the Llama models demonstrate lower semantic similarity with reference summaries (BERTScore) and their performance on LLM-Eval metrics demonstrate an inferior ability to retrieve critical informa

\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline Model & **BertScore-H** & **LLM-Eval-H** & **BertScore-M** & **LLM-Eval-M** & **Human-Eval** \\  & **(\%)** & **(\%)** & **(\%)** & **(\%)** & **(\%)** \\ \hline Claude-3-Sonnet & 88.39 & **73.53** & 88.50 & 69.42 & **69.74** \\ GPT-4o & **91.34** & **72.94** & **90.94** & **70.98** & **67.48** \\ GPT-3.5-turbo & 89.87 & 69.41 & 90.81 & 64.51 & - \\ Llama-3.1-8B-Instruct & 86.44 & 69.41 & 86.44 & 62.35 & 65.87 \\ Llama-3.2-3B-Instruct & 85.60 & 65.10 & 87.09 & 63.73 & 62.75 \\ Gemma-2.9b & 86.00 & 71.37 & 85.48 & **71.57** & 60.83 \\ Human Expert & - & - & - & - & 54.59 \\ Llama3-Med42-8B & **90.56** & 67.45 & **91.34** & 65.29 & 51.11 \\ Phi3-mini-instruct & 88.05 & 57.45 & 89.66 & 60.39 & 50.83 \\ Meditron3-8B & 89.83 & 60.20 & 90.40 & 59.61 & 42.98 \\ \hline \hline \end{tabular}
\end{table}
Table 6: _Summarization performance of various models, evaluated using BERTScore (F1), LLM-Eval, and Human Evaluation. BERTScore-H and LLM-Eval-H represent the metrics when summaries are generated and evaluated using human-generated transcripts, while BERTScore-M and LLM-Eval-M represent the metrics for machine-generated transcripts. Higher is better. Top 2 models in bold._

\begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline Model & positives & negatives & diagnoses & treatment & no hallucination & concise & Total \\ \hline Human Expert & 3.35 & 3.06 & 1.97 & 1.59 & 3.23 & 3.18 & 16.37 \\ \hline Claude-3-sonnet-20240229 & 4.31 & 4.31 & 1.54 & 1.46 & 4.77 & 4.54 & **20.92** \\ GPT-4o & 4.22 & 4.13 & 1.71 & 1.76 & 4.31 & 4.11 & 20.24 \\ Llama-3.1-8B-Instruct & 3.90 & 3.71 & 2.67 & 2.14 & 3.71 & 3.62 & 19.76 \\ Llama-3.2-3B-Instruct & 4.04 & 3.91 & 1.35 & 1.43 & 3.96 & 4.13 & 18.83 \\ Gemma & 3.81 & 3.88 & 1.31 & 1.69 & 3.69 & 3.88 & 18.25 \\ Llama3-Med42-8B & 3.13 & 2.67 & 2.00 & 1.87 & 2.83 & 2.83 & 15.33 \\ Phi-3-mini-4k-instrect & 3.22 & 2.84 & 1.59 & 1.67 & 2.96 & 2.96 & 15.25 \\ Meditron3-8B & 2.86 & 2.48 & 1.46 & 1.23 & 2.45 & 2.41 & 12.89 \\ \hline \hline \end{tabular}
\end{table}
Table 7: Human Evaluation Results showing means of 5-point ratings for each criteria. Higher is better. **Positives: Does the summary include all important positive clinical signs, symptoms, or details in the transcript?**; **Negatives: Does the summary include all important negative clinical signs, symptoms, or details in the transcript?**; **Diagnoses: Does the summary accurately recall the diagnosis from the transcript?**; **Treatment: Does the summary accurately reflect the treatment plan in the transcript?**; **No Hallucination: Does the summary avoid any false- incorrect- or fabricated information?**; **Concies: Is the summary concise- clear- and well-structured?**

\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline Model & **WER** & **Med WER** & **Gen WER** & **AMI** & **Earnings22** & **VoxPop** \\ \hline whisper-medium & 21.27 & 26.49 & **19.47** & 16.6 & - & 7.4 \\ whisper-large-v2 & **20.82** & **23.74** & 19.81 & 16.4 & 12.05 & 7.3 \\ whisper-large-v3 & **20.38** & **23.81** & **19.19** & 16.01 & **11.3** & - \\ whisper-large-v3-turbo & 21.93 & 25.58 & 20.67 & - & - & - \\ distil-whisper-large-v2 & 25.38 & 30.43 & 23.63 & **14.67** & 12.19 & 8.24 \\ distil-whisper-large-v3 & 21.20 & 25.67 & 19.58 & 15.95 & **11.29** & 8.25 \\ parakeet-rnnt-1.1b & 28.16 & 34.03 & 26.13 & 17.1 & 15.15 & **5.44** \\ parakeet-cct-1.1b & 28.97 & 34.16 & 27.19 & 15.67 & 13.75 & 6.56 \\ parakeet-tdt-1.1b & 28.69 & 33.57 & 27.01 & 15.9 & 14.65 & **5.49** \\ canary-lb & 22.82 & 27.40 & 21.25 & **13.53** & 12.05 & 5.79 \\ wav2vec2-large-960h & 86.34 & 88.35 & 81.17 & 37.0 & - & 17.9 \\ mms-lb-all & 61.75 & 69.04 & 59.22 & 42.02 & 31.17 & 17.63 \\ \hline \hline \end{tabular}
\end{table}
Table 5: _WER for various pre-trained models on the entire dataset, including the Medical (Med WER) and General (GenBank WER) portions, as well as AMI [1], Earnings22 [14], and Voxpopuli [21] datasets. Lower is better. Top 2 models in bold._ 

tion from the conversation. The weakest performer overall was Phi3-mini-4k-instruct, particularly in the LLM-Eval scores, signaling significant challenges in medical summarization.

### Human Evaluation

Results in Table 6 show that LLM-as-Judge ratings (LLM-Eval-H) were strongly correlated (Pearsons=0.816) with Human Expert ratings, in line with Zheng et al. (2023).

Table 7 showed that, of the 6 criteria, expert summaries were ranked in the top 3 only at accurately recalling the diagnoses from the transcript, suggesting this was a key focus for experts. LLM summaries were more concise and retained greater amounts of key details when compared to the expert summaries.

The blind expert ratings showed that overall, experts ranked leading LLM summaries higher than reference summaries, scoring consistently higher on the completeness of key facts carried over from the transcript to the summary. This suggests that LLMs may perform better at certain tasks, such as compressing large amounts of information (10+ mins conversations) when compared with time-constrained physicians who may prefer brevity.

### Error Propagation from cascading models

While summarizing ASR transcripts reveal no significant difference in semantic similarity with reference summaries (BERTScore-H vs BERTScore-M), a more nuanced pattern emerges with LLM-Eval-H and LLM-Eval-M metrics where many models show a notable drop in performance. LLVM-Eval-M scores are on average 2 to 5 points lower than LLM-Eval-H. This indicates that ASR errors (noisy transcription outputs) may limit the summarization ability of LLMs leading to loss of critical healthcare information.

## 6 Conclusion

This study highlights several challenges with accented conversational ASR, especially its impact on downstream medical summarization. We benchmark speaker diarization, ASR, and medical summarization, drawing attention to gaps and opportunities to improve accented conversational ASR in the African context.

## 7 Limitations

While simulated conversations provide the opportunity to benchmark ASR systems, they may not fully reflect the complexity of real-world natural dialogues where several interruptions, distractions and background noise are more typical. Additionally, the age range of contributors does not accurately reflect the diversity of real-world speaker populations, potentially impacting the generalizability of the results. The use of general large language models (LLMs) as evaluators may also introduce biases. At 5hrs, the dataset does not reflect the full breadth of medical or general conversations as several domains, subdomains, topics, and medical specialties were not covered in this release. Nonetheless, this work significantly contributes to addressing the gap in ASR systems' adaptability to African accents. Future research should aim to incorporate real-world data, broader age representation, and further optimize models to enhance the robustness and applicability of ASR technologies in diverse African linguistic environments.

## 8 Ethical Considerations

In a world where voice cloning and identity theft are major cybersecurity threats, releasing a voice-based dataset is not without risk. Each recording began with contributors expressing consent to the recording. However, since real names were sometimes used during consent, these segments were removed to protect participants' identities. During doctor-patient conversations, pseudonyms were used to maintain anonymity. Furthermore, releasing real-world doctor-patient conversations risks breaching privacy and confidentiality requirements, hence the preference for simulated conversations.

## 9 Acknowledgment

We thank the Intron Health team for providing the data and compute used in this work and all contributors, patient- and doctor-actors whose voices are included in the dataset.

## References

* B Abacha et al. [2023] Asma Ben Abacha, Wen-wai Yim, Yadan Fan, and Thomas Lin. 2023. An empirical study of clinical note generation from doctor-patient encounters. In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_, pages 2291-2302.
* Abdin et al. [2024] Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakthiari, Hartirat Behl, et al. 2024. Phi-3 technical report: A highly capable language model locally on your phone. _arXiv preprint arXiv:2404.14219_ 

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Atlenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. _arXiv preprint arXiv:2303.08774_.
* Adelani et al. (2022) David Ifeoluwa Adelani, Graham Neubig, Sebastian Ruder, Shruti Rijhwani, Michael Beukman, Chester Palen-Michel, Constantine Lignos, Jesujoba O Alabi, Shamsuddeen H Muhammad, Peter Nabende, et al. 2022. Masakhaner 2.0: Africa-centric transfer learning for named entity recognition. _arXiv preprint arXiv:2210.12391_.
* Afonja et al. (2024) Tejumade Afonja, Tobi Olatunji, Sewade Ogun, Naome A Etori, Abraham Owodunni, and Moshood Yekini. 2024. Performant asr models for medical entities in accented speech. _arXiv preprint arXiv:2406.12387_.
* Askenova et al. (2022) Alena Askenova, Zhehuai Chen, Chung-Cheng Chiu, Daan van Esch, Pavel Golik, Wei Han, Levi King, Bhuvana Ramabhadran, Andrew Rosenberg, Suzan Schwartz, et al. 2022. Accented speech recognition: Benchmarking, pre-training, and diverse data. _arXiv preprint arXiv:2205.08014_.
* Ando and Zhang (2005) Reic Kubota Ando and Tong Zhang. 2005. A framework for learning predictive structures from multiple tasks and unlabeled data. _Journal of Machine Learning Research_, 6:1817-1853.
* Andrew and Gao (2007) Galen Andrew and Jianfeng Gao. 2007. Scalable training of L1-regularized log-linear models. In _Proceedings of the 24th International Conference on Machine Learning_, pages 33-40.
* Anthropic (2023) Anthropic. 2023. The claude 3 model family: Opus, sonnet, haiku. Accessed: 2024-10-15.
* Baevski et al. (2020) Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. _Advances in neural information processing systems_, 33:12449-12460.
* Baye et al. (2020) Johannes Baye, Tesfaye Demeke, Nigusie Birhan, Agumasie Semahegn, and Simon Birknau. 2020. Nurses' work-related stress and associated factors in governmental hospitals in harar, eastern ethiopia: A cross-sectional study. _PloS one_, 15(8):e0236782.
* Bhandari et al. (2024) Nishchal Bhandari, Danny Chen, Miguel Angel del Rio Fernandez, Natalie Delworth, Jennifer Drexler Fox, Miguel Jette, Quinten McNamara, Corey Miller, Ondfej Novotny, Jan Profant, Nan Qin, Martin Rajcizak, and Jean-Philippe Robichaud. 2024. Reverv: Open-source asr and diarization from rev. _Preprint_, arXiv:2410.03930.
* Bharadiya (2023) Jasmin Bharadiya. 2023. A comprehensive survey of deep learning techniques natural language processing. _European Journal of Technology_, 7(1):58-66.
* Biswas and Talukdar (2024) Anjanava Biswas and Wrick Talukdar. 2024. Intelligent clinical documentation: Harnessing generative ai for patient-centric clinical note generation. _arXiv preprint arXiv:2405.18346_.
* Bredin et al. (2020) Herve Bredin, Ruiqing Yin, Juan Manuel Coria, Gregory Gelly, Pavel Korshunov, Marvin Lavechin, Diego Fustes, Hadrien Tietusz, Wassim Bouaziz, and Marie-Philippe Gill. 2020. Pyannote. audio: neural building blocks for speaker diarization. In _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_, pages 7124-7128. IEEE.
* Bredin (2023) Herve Bredin. 2023. pyannote.audio 2.1 speaker diarization pipeline: principle, benchmark, and recipe. In _Proc. INTERSPEECH 2023_.
* Carletta et al. (2005) Jean Carletta, Simone Ashby, Sebastien Bourban, Mike Flynn, Mael Guillemot, Thomas Hain, Jaroslav Kadlec, Vasilis Karaiskos, Wessel Kraaij, Melissa Kronenthal, et al. 2005. The ami meeting corpus: A pre-announcement. In _International workshop on machine learning for multimodal interaction_, pages 28-39. Springer.
* Chen et al. (2022) Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Yanamin Qian, Yao Qian, Jian Wu, Michael Zeng, Xiangzhan Yu, and Furu Wei. 2022. Waylm: Large-scale self-supervised pre-training for full stack speech processing. _IEEE Journal of Selected Topics in Signal Processing_, 16(6):1505-1518.
* Christophe et al. (2024) Clement Christophe, Praveen K Kantihi, Tathagata Raha, Shadab Khan, and Marco AF Pimentel. 2024. Med42-v2: A suite of clinical lllms.
* Chua et al. (2023) Victoria JH Chua, Hexin Liu, Leibiny Paola Garcia Perera, Fei Ting Woon, Jinyi Wong, Xiangyu Zhang, Sanjeev Khudanpur, Andy WH Khong, Justin Dauwels, and Suzy J Styles. 2023. Merlion ccs challenge: A english-mandrin code-switching child-directed speech corpus for language identification and diarization. _arXiv preprint arXiv:2305.18881_.
* Rio et al. (2022) Miguel Del Rio, Peter Ha, Quinten McNamara, Corey Miller, and Shipra Chandra. 2022. Earnings-22: A practical benchmark for accents in the wild. _arXiv preprint arXiv:2203.15591_.
* Desplanques et al. (2020) Brecht Desplanques, Jenthe Thienpondt, and Kris Demupynck. 2020. Ecapa-tdnn: Emphasized channel attention, propagation and aggregation in tdnn based speaker verification. _arXiv preprint arXiv:2005.07143_.
* Doddington et al. (2000) George R Doddington, Mark A Przybocki, Alvin F Martin, and Douglas A Reynolds. 2000. The nist speaker recognition evaluation-overview, methodology, systems, results, perspective. _Speech communication_, 31(2-3):225-254 

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhli Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. _arXiv preprint arXiv:2407.12783_.
* Enavi et al. (2020) Seppo Enavi, Marilisa Amoia, Miguel Del-Agua Teba, Brian Delaney, Frank Diehl, Stefan Hahn, Kristina Harris, Liam McGrath, Yue Pan, Joel Pinto, et al. 2020. Generating medical reports from patient-doctor conversations using sequence-to-sequence models. In _Proceedings of the first workshop on natural language processing for medical conversations_, pages 22-30.
* Etori et al. (2023) Naome Etori, Ebasa Temesgen, and Maria Gini. 2023. What we know so far: Artificial intelligence in african healthcare. _arXiv preprint arXiv:2305.18302_.
* Fareez et al. (2022) Faiha Fatezez, Tishya Parikh, Christopher Wavell, Saba Shahab, Meghan Chevalier, Scott Good, Isabella De Blasi, Rafik Rhouma, Christopher McMahon, Jean-Paul Lam, et al. 2022. A dataset of simulated patient-physician medical interviews with a focus on respiratory cases. _Scientific Data_, 9(1):313.
* Galloway et al. (2024) J Luke Galloway, Dominique Munroe, Pamela D Vohra-Khullar, Christopher Holland, Mary A Solis, Miranda A Moore, and Reema H Dobouk. 2024. Impact of an artificial intelligence-based solution on clinicians' clinical documentation experience: Initial findings using ambient listening technology. _Journal of General Internal Medicine_, pages 1-3.
* Gandhi et al. (2023) Sanchit Gandhi, Patrick von Platen, and Alexander M Rush. 2023. Distil-whisper: Robust knowledge distillation via large-scale pseudo labelling. _arXiv preprint arXiv:2311.00430_.
* Giuseppe et al. (2021) Martucci Giuseppe, Mauro Cettolo, Matteo Negri, and Marco Turchi. 2021. Lexical modeling of asr errors for robust speech translation. In _Proceedings of Interspeech 2021_, pages 2282-2286.
* Hanna and Bojar (2021) Michael Hanna and Ondrej Bojar. 2021. A fine-grained analysis of bertscore. In _Proceedings of the Sixth Conference on Machine Translation_, pages 507-517.
* Harper et al. (2019) Eric Harper, Somshubra Majumdar, Oleksii Kuchaiev, Li Jason, Yang Zhang, E Bakhturina, V Noroozi, S Subramanian, K Nithin, H Jocelyn, et al. 2019. Nemo: A toolkit for conversational ai and large language models. _Computer software), URL: https://github. com/NVIDIA/NeMo_.
* Hinsvark et al. (2021) Arthur Hinsvark, Natalie Delworth, Miguel Del Rio, Quinten McNamara, Joshua Dong, Ryan Westerman, Michelle Huang, Joseph Palakapilly, Jennifer Drexler, Ilya Pirkin, et al. 2021. Accented speech recognition: A survey. _arXiv preprint arXiv:2104.10747_.
* Kanithi et al. (2024) Praveen K Kanithi, Clement Christophe, Marco AF Piemtentel, Tathagata Raha, Nada Saadi, Hamza Javed, Svetlana Maslenkova, Nasir Hayat, Ronnie Rajan, and Shadab Khan. 2024. Medicic: Towards a comprehensive framework for evaluating llms in clinical applications. _arXiv preprint arXiv:2409.07314_.
* 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_, pages 8102-8106.
* Korfiatis et al. (2022) Alex Papadopoulos Korfiatis, Francesco Moramarco, Radmila Sarac, and Aleksandar Savkov. 2022. Primock57: A dataset of primary care mock consultations. _arXiv preprint arXiv:2204.00333_.
* Kuhn (1955) Harold W Kuhn. 1955. The hungarian method for the assignment problem. _Naval research logistics quarterly_, 2(1-2):83-97.
* Landini et al. (2022) Federico Landini, Jan Profant, Mireia Diez, and Lukas Burget. 2022. Bayesian hmm clustering of x-vector sequences (vbx) in speaker diarization: Theory, implementation and analysis on standard tasks. _Computer Speech Language_, 71:101254.
* Le-Duc et al. (2024) Khai Le-Duc, Khai-Nguyen Nguyen, Long Vo-Dang, and Truong-Son Hy. 2024. Real-time speech summarization for medical conversations. _arXiv preprint arXiv:2406.15888_.
* Liu et al. (2023) Yixin Liu, Alexander R Fabbri, Jiawen Chen, Yilun Zhao, Simeng Han, Shafiq Joty, Pengfei Liu, Dragomir Radev, Chien-Sheng Wu, and Arman Cohan. 2023. Benchmarking generation and evaluation capabilities of large language models for instruction controllable summarization. _arXiv preprint arXiv:2311.09184_.
* Mani et al. (2020) Anirudh Mani, Shruti Palaskar, and Sandeep Konam. 2020. Towards understanding asr error correction for medical conversations. In _Proceedings of the first workshop on natural language processing for medical conversations_, pages 7-11.
* Michalopoulos et al. (2022) George Michalopoulos, Kyle Williams, Gagandeep Singh, and Thomas Lin. 2022. Medicalsum: A guided clinical abstractive summarization model for generating medical reports from patient-doctor conversations. In _Findings of the Association for Computational Linguistics: EMNLP 2022_, pages 4741-4749.
* Nguyen-Vo et al. (2022) Hoang-Thi Nguyen-Vo, Huy Nguyen-Gia, Hoan-Duy Nguyen-Tran, Hoang Pham-Minh, Hung Vo-Thanh, and Hao Do-Due. 2022. Marblenet: A deep neural network solution for vietnamese voice activity detection. In _2022 9th NAFOSTED Conference on Information and Computer Science (NICS)_, pages 154-158.
* Ogun et al. (2024) Sewade Ogun, Abraham T Owodunnni, Tobi Olatunji, Enola Alese, Babatunde Oladimeji, Tejumade Afonja, Kayode Daleye, Naome A Etori, and Tosin Adewumi. 2024. 1000 african voices: Advancing inclusive multi-speaker multi-accept speech synthesis. _arXiv preprint arXiv:2406.11727_ 

Tobi Olatunji, Tejumade Afonja, Bonaventure FP Dossou, Atnafu Lambebo Tonja, Chris Chinney Emzeue, Amina Mardiyyah Rufai, and Saibh Singh. 2023a. Arfinamates: Most asr models" butcher" african names. _arXiv preprint arXiv:2306.00253_.
* Olatunji et al. (2023b) Tobi Olatunji, Tejumade Afonja, Aditya Yadavalli, Chris Chinenye Emezue, Sahib Singh, Bonaventrug FD Dossou, Joanne Osuchukwu, Salomev Desei, Atnafu Lambebo Tonja, Naome Erori, et al. 2023b. Affispeech-200: Pan-african accepted speech dataset for clinical and general domain asr. _Transactions of the Association for Computational Linguistics_, 11:1669-1685.
* Panayotov et al. (2015) Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: an asr corpus based on public domain audio books. In _2015 IEEE international conference on acoustics, speech and signal processing (ICASSP)_, pages 5206-5210. IEEE.
* Pasandi and B Pasandi (2022) Hannaneh B Pasandi and Haniyeh B Pasandi. 2022. Evaluation of asr systems for conversational speech: A linguistic perspective. In _Proceedings of the 20th ACM Conference on Embedded Networked Sensor Systems_, pages 962-965.
* Pezik et al. (2022) Piotr Pezik, Gosia Krawentek, Sylwia Karasinska, Pawel Wilk, Paulina Rybinska, Anna Cichosz, Angelika Peljak-Lapinska, Mikolaj Deckert, and Michal Adamczyk. 2022. Diabiz-an annotated corpus of polish call center dialogs. In _Proceedings of the thirteenth language resources and evaluation conference_, pages 723-726.
* Plaza et al. (2021) Miroslaw Plaza, Lukasz Pawlik, and Stanislaw Deniziak. 2021. Call transcription methodology for contact center systems. _IEEE Access_, 9:110975-110988.
* Pratap et al. (2024) Vineel Pratap, Andros Tjandra, Bowen Shi, Paden Tomasello, Arun Babu, Sayani Kundu, Ali Elakhky, Zhaoheng Ni, Apoorv Vyas, Maryam Fazel-Zarandi, et al. 2024. Scaling speech technology to 1,000+ languages. _Journal of Machine Learning Research_, 25(97):1-52.
* Puvvada et al. (2024) Krishna C Puvvada, Piotr Zelasko, He Huang, Oleksii Hrinchuk, Nithin Rao Koluguri, Kunal Dhawan, Somshubra Majumdar, Elena Rastorgueva, Zhehuai Chen, Vitaly Lavrukhin, et al. 2024. Less is more: Accurate speech recognition & translation without web-scale data. _arXiv preprint arXiv:2406.19674_.
* Radford et al. (2023) Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In _International conference on machine learning_, pages 28492-28518. PMLR.
* Rasooli and Tetreault (2015) Mohammad Sadegh Rasooli and Joel R. Tetreault. 2015. Yara parser: A fast and accurate dependency parser. _Computing Research Repository_, arXiv:1503.06733. Version 2.
* Ravanelli and Bengio (2018) Mirco Ravanelli and Yoshua Bengio. 2018. Speaker recognition from raw waveform with sincnet. _2018 IEEE Spoken Language Technology Workshop (SLT)_, pages 1021-1028.
* Ryant et al. (2019) Neville Ryant, Kenneth Church, Christopher Cieri, Alejandrina Cristia, Jun Du, Sriram Ganapathy, and Mark Liberman. 2019. The second dihard diarization challenge: Dataset, task, and baselines. pages 978-982.
* Serikov et al. (2024) Oleg Serikov, Ekaterina Voloshina, Anna Postnikova, Saliha Muradoglu, Eric Le Ferrand, Elena Klyachko, Ekaterina Vylomova, Tatiana Shvarina, and Francis Tyers. 2024. Proceedings of the 3rd workshop on nlp applications to field linguistics (field matters 2024). In _Proceedings of the 3rd Workshop on NLP Applications to Field Linguistics (Field Matters 2024)_.
* Singhal et al. (2023) Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. 2023. Large language models encode clinical knowledge. _Nature_, 620(7972):172-180.
* Skantze (2021) Gabriel Skantze. 2021. Turn-taking in conversational systems and human-robot interaction: a review. _Computer Speech & Language_, 67:101178.
* Spithourakis et al. (2022) Georgios P Spithourakis, Ivan Vulic, Michael Lis, Inigo Casanueva, and Pawel Budzianowski. 2022. Evit: Multilingual spoken dialogue tasks and dataset for knowledge-based enrolment, verification, and identification. _arXiv preprint arXiv:2204.13496_.
* Team et al. (2024) Gemma Team, Morgenac Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Suya Bhuptiaraju, Leonard Hutsenot, Thomas Mesnard, Bobak Shahriari, Alexandre Rame, et al. 2024. Gemma 2: Improving open language models at a practical size. _arXiv preprint arXiv:2408.00118_.
* Temsha et al. (2024) Mohamad-Hani Temsha, Amr Jamal, Khalid Alhasan, Abdulkarim A Temsah, and Khalid H Malki. 2024. Openai o1-preview vs. chatgpt in healthcare: A new frontier in medical ai reasoning. _Cureus_, 16(10).
* Tonja et al. (2024) Atnafu Lambebo Tonja, Israel Abebe Azime, Tadesse Destaw Belay, Mesay Gemeda Yigezu, Moses Ahmed Mehamed, Abinew Ali Ayele, Ebrahim Chekl Jibrli, Michael Melese Woldeydohannis, Olga Kolesnikova, Philipp Slusallek, et al. 2024. EthiolIm: Multilingual large language models for ethiopian languages with task evaluation. _arXiv preprint arXiv:2403.13737_.
* Wang et al. (2021) Changhan Wang, Morgan Reiviere, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, and Emmanuel Dupoux. 2021. VoxPopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint 

Conference on Natural Language Processing (Volume 1: Long Papers), Online. Association for Computational Linguistics.
* Wang et al. (2023) Lucy Lu Wang, Yulia Otmakhova, Jay DeYoung, Thinh Hung Truong, Bailey E Kuehl, Erin Bransom, and Byron C Wallace. 2023. Automated metrics for medical multi-document summarization disagree with human evaluations. _arXiv preprint arXiv:2305.13693_.
* Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, et al. 2020. Transformers: State-of-the-art natural language processing. In _Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations_, pages 38-45.
* Yilmaz et al. (2018) Emre Yilmaz, Astik Biswas, Ewald van der Westhuizen, Febe de Wet, and Thomas Niesler. 2018. Building a unified code-switching asr system for south african languages. _arXiv preprint arXiv:1807.10949_.
* Yim et al. (2023) Wen-wai Yim, Yujuan Fu, Asma Ben Abacha, Neal Snider, Thomas Lin, and Meliha Yetisgen. 2023. Acibench: a novel ambient clinical intelligence dataset for benchmarking automatic visit note generation. _Scientific Data_, 10(1):586.
* Zhang et al. (2019) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. _arXiv preprint arXiv:1904.09675_.
* Zheng et al. (2023) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-jadge with mt-bench and chatbot arena. _Advances in Neural Information Processing Systems_, 36:46595-46623.

## Appendix A Human Evaluation Criteria

This section provides details on the criteria used to evaluate the AI-generated summaries of doctor-patient conversations. Each summary was evaluated based on the following aspects:

1. How well does the summary include all positive clinical signs, symptoms, or details in the transcript?
2. How well does the summary include all negative clinical signs, symptoms, or details in the transcript?
3. How well does the summary recall the diagnosis from the transcript?
4. How accurately does the summary reflect the treatment plan in the transcript?
5. To what degree does the summary avoid false, incorrect, or fabricated information?
6. How concise, clear, and well-structured is the summary?

## Appendix B Prompt Template for the LLM Evaluator

Table 8 outlines the prompt template utilized for the evaluation conducted by OpenAI's o1 model, serving as the LLM evaluator.

## Appendix C Evaluation Metric for the LLM Evaluator

In the LLM evaluation prompt (see Section B), each criterion has a maximum score of 5 points. Since there are six criteria per row, the perfect score for a single row is \(5\times 6=30\) points. For a dataset with ten rows, the total possible score (perfect accuracy) is:

Perfect Accuracy = \(5\times 6\times 10\) = \(300\,\text{points}\).

To compute a model's percentage score for a dataset with N rows:

1. Sum the scores for all six criteria per row.
2. Sum the total scores across all rows.
3. Calculate the percentage score for the model as: 



**LLM as a Judge Prompt Template**

You are a medical expert. Your task is to evaluate the accuracy and quality of an AI-generated summary of a doctor-patient conversation. Provide an impartial assessment based on the criteria below, rating each from 1 to 5 (1 = lowest, 5 = highest). If a specific criterion is not addressed, assign a score of 0.

**Transcript:** {transcript}

**Summary:** {summary}

**Evaluation Criteria:**

1. Completeness of positive clinical signs, symptoms, or relevant details: Does the summary capture all important positive findings?

2. Completeness of negative clinical signs, symptoms, or relevant details: Does the summary capture all important negative findings?

3. Accuracy of diagnosis: Does the summary correctly reflect the diagnosis?

4. Accuracy of treatment plan: Does the summary convey the treatment plan correctly?

5. Factual accuracy: Does the summary avoid false, incorrect, or fabricated information?

6. Clarity and structure: Is the summary concise, clear, and well-structured?

**Please Return the evaluation scores in JSON format:**

\begin{table}
\begin{tabular}{|p{284.9pt}|} \hline
**LLM as a Judge Prompt Template** \\ \hline You are a medical expert. Your task is to evaluate the accuracy and quality of an AI-generated summary of a doctor-patient conversation. Provide an impartial assessment based on the criteria below, rating each from 1 to 5 (1 = lowest, 5 = highest). If a specific criterion is not addressed, assign a score of 0. \\
**Transcript:** {transcript}

**Summary:** {summary}

**Evaluation Criteria:**

1. Completeness of positive clinical signs, symptoms, or relevant details: Does the summary capture all important positive findings?

2. Completeness of negative clinical signs, symptoms, or relevant details: Does the summary capture all important negative findings?

3. Accuracy of diagnosis: Does the summary correctly reflect the diagnosis?

4. Accuracy of treatment plan: Does the summary convey the treatment plan correctly?

5. Factual accuracy: Does the summary avoid false, incorrect, or fabricated information?

6. Clarity and structure: Is the summary concise, clear, and well-structured?

**Please Return the evaluation scores in JSON format:**

\begin{tabular}{l l} \{ & ”Criterion 1”: score, \\  & “Criterion 2": score, \\  & “Criterion 3": score, \\  & “Criterion 4”: score, \\  & “Criterion 5”: score, \\  & “Criterion 6”: score \\ \} \\ \hline \end{tabular}
\end{table}
Table 8: Evaluation task description for assessing AI-generated summaries.



Prompt Template for Summarization

This figure outlines the prompt template utilized for the summarization task.

You are a medical expert; you will be provided a transcript that contains a conversation between a doctor and a patient. The task is for you to summarize the key points in the conversation. **Transcript:** {transcript}

## Appendix E Transcript and Summary Example

This section presents a sample of a human-written transcript in Table 9 and the corresponding LLM-generated summaries for the transcript. While GPT4o captures key details such as pain severity, lifestyle factors, and medical history, its summary omits a vital detail: the patient's experience of rectal bleeding "during urination", which is crucial to the case. The summaries are:

**GPT4o's Summary:** Dr. Philip conducted a consultation with Mrs. Ayike, a 45-year-old woman. She reported experiencing rectal bleeding and abdominal pain for the past two weeks. The pain, rated as a 6 out of 10 on the pain scale, is localized in the lower abdominal region and worsens with pressure. Mrs. Ayike also noted a loss of appetite and fatigue since the onset of her symptoms. She has not sought medical attention prior to this visit and has no history of surgery. She neither smokes nor drinks alcohol. Although Mrs. Ayike does not travel, she revealed a family history of colorectal cancer through her mother. Her usual diet includes "swallow," and she has no food allergies. Mrs. Ayike is married with four children, but currently lives separately from her husband. Dr. Philip concluded the session by acknowledging Mrs. Ayike's details.

**Claude's Summary:** Based on the conversation, the key points are: 1. The patient, Mrs. Ayike, is a 45-year-old woman presenting with rectal bleeding and abdominal pain for the past two weeks. 2. The rectal bleeding occurs when she urinates, and it is not mixed with stool.3. The abdominal pain is rated as 6 out of 10 in severity, and it is localized in the lower abdominal region.4. She has lost her appetite and feels tired since the symptoms started.5. She has not traveled recently and is not a frequent traveler.6. She is married but currently separated from her husband, living in a different state.n7. She has four children.8. She has not undergone any surgery before. 9. She does not smoke or consume alcohol.10. Her mother had colorectal cancer in the past, indicating a family history of the condition.11. Her typical diet includes swallows (a type of food), and she is not allergic to any food. The doctor has gathered essential information about the patient's symptoms, medical history, and lifestyle to aid in the diagnosis and treatment process.

**Gamma's Summary:** Mrs. Ayike, a 45-year-old women, presented to Dr. Philip complaining of two weeks of abdominal pain and rectal bleeding. Key points:Symptoms:Lower abdominal pain rated 6/10, worsening with pressure. Rectal bleeding during urination, not mixed with stool. Loss of appetite and fatigue. Medical History: No previous surgeries. No smoking or alcohol consumption. Family history of colorectal cancer (mother). Lifestyle: Prefers a diet high in swallow. No known food allergies. Not a traveler. Separated from her husband, living with four children. Next steps: Based on the symptoms and family history, Dr. Philip will likely order further investigations to determine the cause of Mrs. Ayike's abdominal pain and rectal bleeding. This may include blood tests, stool tests, imaging studies (such as a colonoscopy or CT scan), and possibly a biopsy.

**Med42-8B's summary:** Mrs. Ayike, a 45-year-old female, presents with rectal bleeding and lower abdominal pain, rated 6/10 in severity, which started two weeks ago. She reports seeing red blood in her urine but not mixed with stool. She has not sought medical attention previously for these symptoms. She describes her appetite as poor and feels tired. She denies any recent travel, smoking, or alcohol use. She is married with four children but is currently separated from her husband. Her mother had colorectal cancer. She consumes a diet primarily consisting of fufu (swallow). There are no known food allergies.

**GPT3.5's summary:** Mrs. Aike, a 45-year-old patient, visited Dr. Philip due to rectal bleeding and abdominal pain that started two weeks ago. The abdominal pain is severe, rated at a 6 on a scale of 1 to 3, and is localized in the lower abdominal region. She has also experienced a loss of appetite, fatigue, and difficulty eating since the pain started. Mrs. Aike is married with four children, not living with her husband, and 

has no history of surgeries, smoking, or alcohol consumption. Her mother had colorectal cancer before, indicating a family history of similar issues. She prefers eating swallow and is not allergic to any food. This is her first visit to seek medical help for these symptoms.

**Meditron3-8B's summary:** The patient, Mrs. Ayike, is a 45-year-old woman who has been experiencing rectal bleeding and abdominal pain for two weeks. She has not undergone any surgery before and does not smoke or take alcohol. Her mother had colorectal cancer before. She has lost her appetite and feels tired. The doctor asks her about her family history, food allergies, and the severity of her symptoms. The doctor also asks about her marital status and children. The patient's symptoms are similar to those of colorectal cancer, which her mother had.

**Llama-3.1-8B-Instruct's summary:** Based on the provided transcript, here are the key points from the conversation between Dr. Philip and Mrs. Ayike: 1. Patient's Introduction: Mrs. Ayike, a 45-year-old woman, presents to the doctor with rectal bleeding and abdominal pain. 2.Symptoms: The abdominal pain started two weeks ago, and she experiences rectal bleeding when urinating, but not mixed with stool. The pain is located in the lower abdominal region and worsens with

\begin{table}
\begin{tabular}{|p{207.6pt}|} \hline
**Transcript** \\ \hline
**Doctor:** Good afternoon. My name is Dr. Philip, and what’s your name? \\ \hline
**Patient:** My name is Mrs. Ayike. \\ \hline
**Doctor:** Okay, how old are you? \\ \hline
**Patient:** I am 45 years old. \\ \hline
**Doctor:** Okay. You’re welcome, my dear. What brought you here? \\ \hline
**Patient:** I have rectal bleeding and abdominal pain. \\ \hline
**Doctor:** Can you tell me more about it? \\ \hline
**Patient:** The pain started two weeks ago, and the rectal bleeding happens when I urinate. The blood is not mixed with stool. \\ \hline
**Doctor:** Have you gone for any check-ups? \\ \hline
**Patient:** No, this is my first time coming here. \\ \hline
**Doctor:** How severe is the abdominal pain, from a scale of 1 to 10? \\ \hline
**Patient:** 6. \\ \hline
**Doctor:** Which side of the abdominal region do you feel the pain? \\ \hline
**Patient:** I feel pain in my lower abdominal region, and it worsens with pressure. And since the last two weeks, as I’ve started the abdominal pain, I have not been able to eat well. I lost appetite. I feel tried. \\ \hline
**Doctor:** Have you traveled recently? \\ \hline
**Patient:** No, I don’t travel. \\ \hline
**Doctor:** Are you married? \\ \hline
**Patient:** Yes, I am married, with four children, but my husband lives in another state. \\ \hline
**Doctor:** Have you undergone any surgery before? \\ \hline
**Patient:** No. \\ \hline
**Doctor:** Do you smoke or drink alcohol? \\ \hline
**Patient:** No. \\ \hline
**Doctor:** Does anyone in your family have similar issues? \\ \hline
**Patient:** Yes, my mother had colorectal cancer. \\ \hline
**Doctor:** What foods do you normally eat? \\ \hline
**Patient:** I like taking swallow. I like swallow a lot. \\ \hline
**Doctor:** Are you allergic to any food? \\ \hline
**Patient:** No. \\ \hline \end{tabular}
\end{table}
Table 9: Snippet of a doctor-patient conversation (transcript) written by a clinician 

pressure. 3. Medical History: Mrs. Ayike has not undergone any surgery before and does not have a history of smoking or alcohol consumption. 4. Family History: Her mother had colorectal cancer before.5. Diet and Allergies: Mrs. Ayike likes eating swallow (a type of food) and is not allergic to any food. 6.Lifestyle : She is married with four children but is currently separated from her husband. She does not travel and has not been able to eat well or feel energetic due to her symptoms. 7. Current Condition: Mrs. Ayike rates her abdominal pain as a 6 out of 10 and has lost appetite and feels tired since the onset of symptoms. Based on the information provided, Dr. Philip will likely perform a thorough examination and possibly order diagnostic tests to determine the cause of Mrs. Ayike's symptoms. Given her family history of colorectal cancer, a colonoscopy may be recommended to rule out any potential malignancies.

**Llama-3.2-3B-Instruct's summary:** Here's a summary of the key points in the conversation between Dr. Philip and Mrs. Ayike: Patient's Symptoms: Rectal bleeding (not mixed with stool), Abdominal pain (rated 6 out of 10) in the lower abdominal region, Loss of appetite and tiredness for the past two weeks. Medical History: No previous surgeries, No smoking or alcohol consumption, No known allergies to food Family History: Mother had colorectal cancer before. Lifestyle and Habits: Married with four children, but not living with her husband. No history of traveling. Prefers swallowing food. Other Information: Mrs. Ayike is 45 years old. She has not been able to eat well due to the abdominal pain. She is seeking medical attention for the first time, as the symptoms started two weeks ago.

**Phi3-mini-instruct's summary:** Dr. Philip, a medical expert, is speaking with Mrs. Ayike, a 45-year-old woman who has been experiencing rectal bleeding and abdominal pain for two weeks. Mrs. Ayike has not had any check-ups for her symptoms and has lost her appetite, feeling tired. She has not traveled recently and is not a smoker. She does not consume alcohol and has no known food allergies. Mrs. Ayike's mother had colorectal cancer. Dr. Philip asks about her diet, which mainly consists of swallow, and inquires about any family history of similar issues.

## Appendix F Overlapping Speech Example

In this example, overlapping speech occurs between Speaker 1 and Speaker 2. Speaker 2's "No" overlaps with Speaker 1's utterance starting with "Ok, thank you very much." Both utterances are fully captured with precise timestamps to ensure accurate representation of the conversation. This example illustrates how overlaps are handled in the transcription process for the dataset.
 

Figure 4: Summarization results for several LLMs

Figure 3: Comparison of Medical and General WER for Different Models





\begin{table}
\begin{tabular}{|c|l|l|} \hline
**Timestamp** & **Speaker** & **Utterance** \\ \hline
02:40.98 – 02:41.99 & Speaker 1 & Past hospital admission? \\ \hline
02:42.97 – 02:43.02 & Speaker 2 & No. \\ \hline
02:43.00 – 02:45.99 & Speaker 1 & Ok, thank you very much. Do you live alone? \\ \hline \end{tabular}
\end{table}
Table 10: Example of overlapping speech with accurate timestamps and speaker annotations.

Figure 5: WER, Med WER, and Non-Med WER for Various Models