Exploring Speaker-Related Information in Spoken Language Understanding for Better Speaker Diarization
Year: 2023

Abstract:
Speaker diarization(SD) is a classic task in speech processing and is crucial in multi-party scenarios such as meetings and conversations. Current mainstream speaker diarization approaches consider acoustic information only, which result in performance degradation when encountering adverse acoustic conditions. In this paper, we propose methods to extract speaker-related information from semantic content in multi-party meetings, which, as we will show, can further benefit speaker diarization. We introduce two sub-tasks, Dialogue Detection and Speaker-Turn Detection, in which we effectively extract speaker information from conversational semantics. We also propose a simple yet effective algorithm to jointly model acoustic and semantic information and obtain speaker-identified texts. Experiments on both AISHELL-4 and AliMeeting datasets show that our method achieves consistent improvements over acoustic-only speaker diarization systems.

heading: 1 Introduction
text: Speaker diarization(SD) is the task of answering the question “who speaks when" by partitioning audio into segments with speaker identities. In most application settings, the results of speaker diarization are perceived by readers through the assignment of speaker labels to the corresponding words or sentences transcribed from an Automatic Speech Recognition(ASR) system.
Despite the rich profusion of transcribed texts, mainstream speaker diarization systems consider only acoustic information (Park et al., 2021; Horiguchi et al., 2020; Park et al., 2020; Fujita et al., 2019; Zheng et al., 2021; Du et al., 2022a). Traditional SD systems usually consist of the following components: (1) Voice activity detection (VAD) to filter out non-speech frames. (2) Extraction of speaker embeddings from the short audio segments, using popular models such as i-vector(Dehak et al.,
2011), d-vector(Zhang and Koishida, 2017) and xvector(Snyder et al., 2018). (3) Clustering embeddings into several classes using algorithms such as agglomerative hierarchical clustering (AHC) (Day and Edelsbrunner, 1984), spectral clustering(SC) (Wang et al., 2017), and HDBSCAN (Zheng et al., 2022). Various speaker embedding model and clustering methods have been explored and proposed in (Yu et al., 2021; Dawalatabad et al., 2021; He et al., 2021; Zheng and Suo, 2022; Du et al., 2022b).
Utilizing only acoustic information has significant limitations. For example, the performance of SD system suffers from obvious degradation in adverse acoustic conditions such as noise, reverberation, and far-field recordings. In addition, we often encounter speakers with similar voice characteristics, which pose serious challenge to clustering them into expected classes. Given the abundance of transcribed texts present in meetings and conversations, it is of sufficient interest to explore the possibilities of utilizing semantic information to go beyond the limits of acoustic-only speaker diarization.
Some previous works tried to use semantic information to classify roles in two-speaker conversations, such as doctor-patient conversation and pilotair traffic controller dialogue(Zuluaga-Gomez et al., 2021; Flemotomos and Narayanan, 2022). However, these methods are only suitable for specific two-speaker scenarios where the roles are clearlydefined, such as medical diagnosis, job interviews, and air traffic communications. In this work we focus on open multi-party meeting scenarios where the number of speakers is unknown and the relations between speakers are unspecified.
Speaker identity information has been proven to be beneficial to many downstream NLP tasks(Chi et al., 2017a; Zhang et al., 2018; Chi et al., 2017b). However, these works only consider speaker identities as given ground truth (Carletta et al., 2006; Janin et al., 2003; Zhong et al., 2021), which is
14068
impractical in real world settings. Therefore, it is crucial to make valid inference of speaker identities on the transcribed conversations using a wellperformed speaker diarization system.
The main contributions of this paper include: (1) We propose two semantic tasks to extract speaker-related information from automatically transcribed texts, namely Dialogue Detection and Speaker-Turn Detection.
(2) We design a simple yet effective integration method to effectively combine semantic and acoustic information for more robust speaker diarization.

heading: 2 Proposed Methods

heading: 2.1 A Novel Multi-modal Framework
text: Figure 1 illustrates the proposed semantic-acoustic speaker diarization system, along with its relation with upstream ASR components and downstream NLP applications. We introduce a Spoken Language Processin(SLP) module involving two subtasks to extract speaker-related information from transcribed texts. The acoustic-based speaker diarization system is used to process original audio, perform segmentation and estimate speaker embeddings for each segments. To associate speaker embeddings with corresponding text phrases, a forced alignment component was introduced to our system. Finally, we propose an integration method to collectively process outputs from SLP module, acoustic SD module, and forced alignment module.

heading: 2.2 Learning Speaker Information From Texts
text: To extract semantic speaker-related information, we define two sub-tasks: dialogue detection and speaker-turn detection.
Dialogue detection takes a sequence of sentences as input and determines whether this is transcribed from a multi-speaker dialogue or a single-speaker speech. Dialogue-detection can be defined as a binary classification problem.
Speaker turn detection tries to predict, for each given sentence in the sequence, the probability of the occurrence of speaker change. Speaker turn detection can be defined as a sequence labeling problem, where the goal is to determine whether the given position represents a point of change in speaker role from a semantic perspective.
Both dialogue detection and speaker turn detection models are fine-tuned from a pre-trained BERT language model. Design of training samples and details of experiments are discussed in next section.

heading: 2.3 Integrating Semantic-Acoustic Information
text: In this section we describe how speaker-related information extracted from semantic content can assist us in improving upon acoustic-only SD system. A traditional SD system typically involves an audio segmentation module and an embedding clustering module. Poor segmentation and incorrect clustering are the most common problems in speaker diarization. Semantic information from
dialogue detection helps improve clustering accuracy and speaker turn detection helps to find more precise place in text where a change of speaker occurs.
Note that dialogue detection and speaker turn detection tasks can be solved either by acoustic-only approach or semantic-only approach. Semanticonly approach is described above. Acoustic-only results can be derived directly from acoustic-based speaker clustering. The speaker clustering algorithm assigns a cluster label to each speakersegment. Acoustic results for dialogue detection can be obtained simply by checking whether the number of different speaker labels is larger than 1. Results for speaker turn detection can be obtained by analyzing the transition patterns of speakersegment labels or predicting change points using an acoustic-based neural networks such as TargetSpeaker VAD(He et al., 2021).
Semantic-Acoustic Dialogue Detection. Let z(s) denote the result of binary classification output of semantic dialogue detection and z(a) be the counterpart of acoustic dialogue detection. We also define Dp to be the distance of the largest speaker cluster present in the dialogue to its furthest cluster, and Dq to be the standard deviation of the cosine distances among all speaker embeddings present in the selected speech. Dp measures how spread out different clusters are and Dq measures how tight embeddings in one cluster are grouped together. Then the fusion score ŝ is estimated by:
ŝ = z(a)z(s)+z(a)(ps+α1Dp)+z (s)(ps+α2Dq),
(1) where α1 and α2 are learnable and ps is logit output from semantic dialogue detection. For some threshold θ, the binary output of semantic-acoustic dialogue detection is represented by the indicator function:
ẑfusiondd = 1ŝ>θ (2)
Once semantic-acoustic dialogue detection obtain results for all sentence sequences that cover the entire transcribed meeting, we re-adjust the acoustic-based clustering results. By doing this we are able to incorporate semantic information to improve speaker clustering. More details can be found in Appendix B.
Semantic-Acoustic Speaker Turn Detection. Semantic-only speaker turn detection outputs a sequence of probability of the occurrence of speaker
change. Let pn be the probability at position n, and qn represents the speaker change probability from an acoustic-only model near position n. qn is obtained by taking the maximum probability of the closest 200 frames estimated by the Target-Speaker VAD model. Then the integrated speaker-change probability is given by
p̃n = β1pn + β2qn (3)
for some learnable hyperparameters β1 and β2. Boundary and Outlier Correction. We use semantic information to correct boundary errors caused by errors and mismatches from the forcedalignment and ASR models. We also use semantic information to correct outliers in embedding extraction. To improve system robustness, we exclude audio segments that are too short from clustering. Outliers and left-out embeddings are assigned to the closest cluster.

heading: 3 Experiments and Results

heading: 3.1 Datasets
text: We conduct experiments on AISHELL-4(Fu et al., 2021) and AliMeeting(Yu et al., 2022) datasets. Both focus on multi-party meeting scenario, where all speech content are manually annotated. Table 1 listed detailed information about the datasets. We perform experiments using both ground truth (GT) text and text transcribed from ASR system.

heading: 3.2 Experimental Setups
text: In our experiments, the acoustic modules, including ASR, ASR Post-Processing, Embedding Extractor, and Forced Alignment models, are fixed and used consistently throughout all our experiments. In details, the ASR system we introduced was based on UniASR(Gao et al., 2020). The ASR Post-Processing contained PunctuationPrediction(Chen et al., 2020) and Text-Smoothing which are common used in meeting scenrio. The Forced Alignment module we introduced was
based on (McAuliffe et al., 2017). For acoustic speaker diarization system, we employed a speaker embedding extractor based on ECAPATDNN(Desplanques et al., 2020), while for speaker clustering, we utilized Spectral Clustering algorithm with p-percentile(Wang et al., 2017).
We fine-tune the semantic models for dialogue detection and speaker turn detection tasks based on the pre-trained BERT language model1 using the text from AISHELL-4 and AliMeeting training sets. Training samples are sequences of sentences generated by a sliding-window method with a window length of 64 and a shift of 16 and the label for these two semantic subtasks can be generated by the speaker label from the speech content manually annotated in the datasets.
For dialogue detection task, we fine-tune for 3 epochs on train dataset with a learning rate of 5e-6 and a batch size of 64. For speaker-turn detection task, we also fine-tune for 3 epochs on train dataset with a learning rate of 1e-6 and a batch size of 64.
1Based on bert-base-chinese from HuggingFace

heading: 3.3 Results and Discussions
text: We compare our proposed methods with the classic speaker diarization system mentioned in Section 1.
Table 2 shows the results of dialogue detection and speaker-turn detection tasks from acousticonly, semantic-only, and multi-modal models. We not only compare results using ASR-transcribed texts, but also conduct experiments using ground truth texts as inputs, in order to see the optimal improvements introduced by semantic information. The multi-modal model surpasses single-modal results on both GT and ASR text. The experiments demonstrate that semantic model can effectively supplement acoustic-only model, resulting in more precise speaker representation. It is expected that the introduction of semantic information on ASR text does not result in a significant improvement due to the a lot of errors present in the text. However, our multi-modal approach shows consistent improvement in both GT and ASR-based results, indicating the robustness of our method.
We use the cp-WER metric to measure the speaker diarization task. We introduce a new metric speaker-WER that aims to measure the word error rate caused solely by speaker errors. More
details about the metrics can be found in Appendix A. Table 3 shows the final results of our speaker diarization system. Compared to the baseline, which only uses acoustic information, our system incorporating semantic information shows a significant improvement. The results for “Semantic-acoustic info. - A" indicate that only semantic information is used for sub-tasks Dialogue Detection and Speaker Turn Detection, while “Semantic-acoustic info. - B" indicates that both semantic and acoustic information are used in the two sub-tasks.

heading: 4 Conclusions
text: We propose a novel multi-modal speaker diarization system that utilizes two spoken language processing tasks, dialogue detection and speaker-turn detection, to extract speaker-related information from text. These information are then combined with acoustic information to improve the overall performance of speaker diarization. Our experiments demonstrate that incorporating semantic information can effectively address the limitations of single-modal speech.

heading: 5 Limitations
text: The performance of SLP tasks rely heavily on the accuracy of ASR system. Poorly-transcribed texts can lead to degradation of our multi-modal method. Since we cannot easily obtain accurate speaker-turn information from the ASR text, the training set for SLP tasks based on ASR text is also not easy to obtain. In future work, we will try more methods, like Data Arguments, to get better results on ASR text.
Overlapping speech is another challenge for the task, as a monaural ASR system can no longer capture all spoken words from all speakers. In future work, we plan to explore methods such as speech separation or multi-party ASR to handle overlapping speech.

heading: A Metrics
text: The performance metrics for speaker diarization is concatenated minimum-permutation Word error rate(cpWER). The cpWER is computed as follows:
1. Concatenate each speaker’s utterances both from reference and hypothesis results.
2. Compute the WER between the reference and all possible speaker permutation of hypothesis.
3. Choose the lowest WER among the results from all the speaker permutations as the final cpWER.
Specifically, the Word Error Rate(WER) is calculated by:
Ewer = NIns +NSubs +NDel
NTotal × 100% (4)
The cpWER is affected both by the speaker diarization system and speech recognition system.
Note that the number of speakers in the system result and the reference result are not equal. If we ignore the cpWER measured by this part of the text, we will record it as Ecp−matched. If we think that this part of the text should be considered all wrong, we will record the cpWER this time for Ecp−all.
We know that WER calculates the minimum edit distance from the system result to the reference result. In the calculation process, the cost of changing one sequence into another sequence by using three operations of insertion, deletion and replacement is counted, while the calculation of cpWER additionally introduces the operation of modifying the speaker ID of a word to make two speaker-labeled texts become consistent.
Since the speaker diarization system cannot modify the text results of speech recognition, we calculate the speaker-WER by removing the errors caused by ASR results from cpWER. Compared to the three string operations in WER, we additionally define an operation to convert one text result with speaker ID to another by modifying the speaker ID of a word. Similar to the WER algorithm, we use dynamic programming to count the number of operations for changing the speaker ID. The speaker-WER results is calculated by:
EspeakerWER = NSpk−Cost NTotal
(5)

heading: B Pseudocodes to Update Speaker Diarization from Semantic results
text: The following psuedocodes show how results from Dialogue Detection and Speaker Turn Detection are utilized to re-adjust and update speaker diarization results.
We create a sequence of speaker change occurrences, Pstp = {p1, p2, ..., pN}, where N is the number of speaker changes, by combining the results of the Dialogue Detection and Speaker-Turn Detection tasks.
We propose a split process, as shown in Algorithm 1, to adjust the speaker IDs of certain segments to align the cluster results with the speaker change results as closely as possible by increasing the number of speakers appropriately.
Algorithm 1 The Semantic Split Process Require: Pstp = {p1, p2, ..., pN} The set of the
speaker turn point, pi ∈ {0, 1} D = {d1, d2, ..., dN} Each text segments divided by speaker change, E = {e1, e2, ..., eN} Set of mean embedding belong to dk, τsplit The split threshold
Ensure: Ŝ ← [1], B ← [e1], Nspk ← 1, i← 2 while i ≤ N do
dist = f(ei, B) ∈ RSize(B) if pi = 1 then
if dist < τsplit then si = argminsi∈S dist Ŝ ← Ŝ.append(si) else Nspk ← Nspk + 1 si ← Nspk Ŝ ← Ŝ.append(si)
end if else
si ← si−1 Ŝ ← Ŝ.append(si)
end if B ← B ∪ ei
end while
After the split process, a merge process is implemented to eliminate redundant speaker IDs. We consider both acoustic information, such as the similarity distance between speaker embeddings, and semantic information, such as the score differences between the merge of speakers i and j, computed from the Dialogue-Detection and Speaker-Turn Detection results of the utterances with the speaker IDs in two adjacent text segments. The pseudocode for the merge process is outlined in Algorithm 2.
Algorithm 2 The Semantic Merge Process Require: E Set of mean embedding of all the
speakers S Set of speakers after split process τmerge the merge threshold while Can Not Merge do
costsim ← cosine_distance(E) costdd ← scoreafters − scorebefores imerge, jmerge = argmini,j∈S(costsim +
costs) if costall[imerge, jmerge] < τmerge then
update S by merge speaker imerge, jmerge update E by merge speaker imerge, jmerge
end if end while
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section 5

heading: A2. Did you discuss any potential risks of your work?
text: Not applicable. Left blank.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.

heading: B1. Did you cite the creators of artifacts you used?

heading: No response.

heading: B2. Did you discuss the license or terms for use and / or distribution of any artifacts?

heading: No response.

heading: B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.

heading: B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.

heading: B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be.

heading: No response.
text: C 3 Did you run computational experiments? Section 3

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 3
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 3

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 3
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.

heading: D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? No response.

heading: D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.

heading: D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? No response.

heading: D4. Was the data collection protocol approved (or determined exempt) by an ethics review board?

heading: No response.

heading: D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: No response.

heading: Future Work
text: In future work, we will try more methods, like Data Arguments, to get better results on ASR text. Overlapping speech is another challenge for the task, as a monaural ASR system can no longer capture all spoken words from all speakers. In future work, we plan to explore methods such as speech separation or multi-party ASR to handle overlapping speech