Abstract: This paper describes a system that generates speaker-annotated transcripts of meetings by using a microphone array and a 360-degree camera. The hallmark of the system is its ability to handle overlapped speech, which has been an unsolved problem in realistic settings for over a decade. We show that this problem can be addressed by using a continuous speech separation approach. In addition, we describe an online audio-visual speaker diarization method that leverages face tracking and identification, sound source localization, speaker identification, and, if available, prior speaker information for robustness to various real world challenges. All components are integrated in a meeting transcription framework called SRD, which stands for "separate, recognize, and diarize". Experimental results using recordings of natural meetings involving up to 11 attendees are reported. The continuous speech separation improves a word error rate (WER) by 16.1% compared with a highly tuned beamformer. When a complete list of meeting attendees is available, the discrepancy between WER and speaker-attributed WER is only 1.0%, indicating accurate word-to-speaker association. This increases marginally to 1.6% when 50% of the attendees are unknown to the system.

Introduction:  The goal of meeting transcription is to have machines generate speaker-annotated transcripts of natural meetings based on their audio and optionally video recordings. Meeting transcription and analytics would be a key to enhancing productivity as well as improving accessibility in the workplace. It can also be used for conversation transcription in other domains such as healthcare BIBREF0. Research in this space was promoted in the 2000s by NIST Rich Transcription Evaluation series and public release of relevant corpora BIBREF1, BIBREF2, BIBREF3. While systems developed in the early days yielded high error rates, advances have been made in individual component technology fields, including conversational speech recognition BIBREF4, BIBREF5, far-field speech processing BIBREF6, BIBREF7, BIBREF8, and speaker identification and diarization BIBREF9, BIBREF10, BIBREF11. When cameras are used in addition to microphones to capture the meeting conversations, speaker identification quality could be further improved thanks to the computer vision technology. These trends motivated us to build an end-to-end audio-visual meeting transcription system to identify and address unsolved challenges. This report describes our learning, with focuses on overall architecture design, overlapped speech recognition, and audio-visual speaker diarization. When designing meeting transcription systems, different constraints must be taken into account depending on targeted scenarios. In some cases, microphone arrays are used as an input device. If the names of expected meeting attendees are known beforehand, the transcription system should be able to provide each utterance with the true identity (e.g., “Alice” or “Bob”) instead of a randomly generated label like “Speaker1”. It is often required to show the transcription in near real time, which makes the task more challenging. This work assumes the following scenario. We consider a scheduled meeting setting, where an organizer arranges a meeting in advance and sends invitations to attendees. The transcription system has access to the invitees' names. However, actual attendees may not completely match those invited to the meeting. The users are supposed to enroll themselves in the system beforehand so that their utterances in the meeting can be associated with their names. The meeting is recorded with an audio-visual device equipped with a seven-element circular microphone array and a fisheye camera. Transcriptions must be shown with a latency of up to a few seconds. This paper investigates three key challenges.  Speech overlaps: Recognizing overlapped speech has been one of the main challenges in meeting transcription with limited tangible progress. Numerous multi-channel speech separation methods were proposed based on independent component analysis or spatial clustering BIBREF12, BIBREF13, BIBREF14, BIBREF15, BIBREF16, BIBREF17. However, there was little successful effort to apply these methods to natural meetings. Neural network-based single-channel separation methods using techniques like permutation invariant training (PIT) BIBREF18 or deep clustering (DC) BIBREF19 are known to be vulnerable to various types of acoustic distortion, including reverberation and background noise BIBREF20. In addition, these methods were tested almost exclusively on small-scale segmented synthetic data and have not been applied to continuous conversational speech audio. Although the recently held CHiME-5 challenge helped the community make a step forward to a realistic setting, it still allowed the use of ground-truth speaker segments BIBREF21, BIBREF22. We address this long-standing problem with a continuous speech separation (CSS) approach, which we proposed in our latest conference papers BIBREF23, BIBREF24. It is based on an observation that the maximum number of simultaneously active speakers is usually limited even in a large meeting. According to BIBREF25, two or fewer speakers are active for more than 98% of the meeting time. Thus, given continuous multi-channel audio observation, we generate a fixed number, say $N$, of time-synchronous signals. Each utterance is separated from overlapping voices and background noise. Then, the separated utterance is spawned from one of the $N$ output channels. For periods where the number of active speakers is fewer than $N$, the extra channels generate zeros. We show how continuous speech separation can fit in with an overall meeting transcription architecture to generate speaker-annotated transcripts. Note that our speech separation system does not make use of a camera signal. While much progress has been made in audio-visual speech separation, the challenge of dealing with all kinds of image variations remains unsolved BIBREF26, BIBREF27, BIBREF28.  Extensible framework: It is desirable that a single transcription system be able to support various application settings for both maintenance and scalability purposes. While this report focuses on the audio-visual setting, our broader work covers an audio-only setting as well as the scenario where no prior knowledge of meeting attendees is available. A modular and versatile architecture is desired to encompass these different settings. To this end, we propose a framework called SRD, which stands for “separate, recognize, and diarize”, where CSS, speech recognition, and speaker diarization takes place in tandem. Performing CSS at the beginning allows the other modules to operate on overlap-free signals. Diarization is carried out after speech recognition because its implementation can vary significantly depending on the application settings. By choosing an appropriate diarization module for each setting, multiple use cases can be supported without changing the rest of the system. This architecture also allows transcriptions to be displayed in real time without speaker information. Speaker identities for each utterance may be shown after a couple of seconds.  Audio-visual speaker diarization: Speaker diarization, a process of segmenting input audio and assigning speaker labels to the individual segments, can benefit from a camera signal. The phenomenal improvements that have been made to face detection and identification algorithms by convolutional neural networks (CNNs) BIBREF29, BIBREF30, BIBREF31 make the camera signal very appealing for speaker diarization. While much prior work assumes the batch processing scenario where the entire meeting recording can be processed multiple times, several studies deal with online processing BIBREF32, BIBREF33, BIBREF34, BIBREF35. However, no previous studies comprehensively address the challenges that one might encounter in real meetings. BIBREF32, BIBREF33 do not cope with speech overlaps. While the methods proposed in BIBREF34, BIBREF35 address the overlap issue, they rely solely on spatial cues and thus are not applicable when multiple speakers sit side by side. Our diarization method handles overlapping utterances as well as co-located speakers by utilizing the time-frequency (TF) masks generated by CSS in speaker identification and sound source localization (SSL). In addition, several enhancements are made to face identification to improve robustness to image variations caused by face occlusions, extreme head pose, lighting conditions, and so on.

Question: Are face tracking, identification, localization etc multimodal inputs in some ML model or system is programmed by hand?
Answer: Input in ML model