Patent Description:
Speaker diarization is the process of partitioning an input audio stream into homogenous segments according to speaker identity. In an environment with multiple speakers, speaker diarization answers the question "who is speaking when" and has a variety of applications including multimedia information retrieval, speaker turn analysis, and audio processing to name a few In particular, speaker diarization systems are capable of producing speaker boundaries that have the potential to significantly improve acoustic speech recognition accuracy.

<CIT> describes a method for facilitating a remote conference that includes receiving a digital video and a computer-readable audio signal. A face recognition machine is operated to recognize a face of a first conference participant in the digital video, and a speech recognition machine is operated to translate the computer-readable audio signal into a first text. An attribution machine attributes the text to the first conference participant. A second computer-readable audio signal is processed similarly, to obtain a second text attributed to a second conference participant. A transcription machine automatically creates a transcript including the first text attributed to the first conference participant and the second text attributed to the second conference participant.

<CIT>describes a computer-implemented method that includes identifying a plurality of protected pieces from a conversation. The computer-implemented method further includes generating one or more confidence scores for each protected piece, wherein a confidence score is a degree of associativity between a protected piece and a type of sensitive information. The computer-implemented method further includes determining that the protected piece is associated with the type of sensitive information. The computer-implemented method further includes determining a type of protection action for each protected piece in the plurality of protected pieces. The computer-implemented method further includes performing the type of protection action for each protected piece in the plurality of protected pieces to form a modified conversation that is devoid of the sensitive information. A corresponding computer system and computer program product are also disclosed.

<NPL> presents a condensed survey of existing research and technologies, including smart meeting system architecture, meeting capture, meeting recognition, semantic processing, and evaluation methods.

<CIT>describes a method or system for enabling training of a language model which includes producing segments of text in a text corpus and counts corresponding to the segments of text, the text corpus being in a depersonalized state. The method further includes enabling a system to train a language model using the segments of text in the depersonalized state and the counts. Because the data is depersonalized, actual data may be used.

<CIT> describes methods and an apparatus for providing privacy during video or other communication across a network. In one configuration, video conferencing is performed over a network between two or more users. Images are generated by digital video cameras and processed by video servers. The digital video camera generates one or more digital images that are processed by the processing server, including identifying and obstructing any artifacts (e.g., faces, hands, etc.) in the images. The processing also optionally includes the tracking of the artifacts in the images as they move within the image, as well as search for new faces that may enter the field of view In another configuration of the invention, video conferencing is performed over a network between two or more users. Images are generated by digital video cameras and processed by video servers. During the videoconference, one or more users may select a video (and audio) muting mode during which any artifacts of interest in the images (or portions thereof) are identified and obscured.

<CIT> describes a method executed by a computer that includes receiving an image from a client device. A facial recognition technique is executed against an individual face within the image to obtain a recognized face. Privacy rules are applied to the image, where the privacy rules are associated with privacy settings for a user associated with the recognized face. A privacy protected version of the image is distributed, where the privacy protected version of the image has an altered image feature.

One aspect of the disclosure provides a method for generating a privacy-aware meeting room transcript from a content stream. The method includes receiving, at data processing hardware, an audio-visual signal comprising audio data and image data. The audio data corresponds to speech utterances from a plurality of participants in a speech environment and the image data represents faces of the plurality of participants in the speech environment. The method also includes receiving, at the data processing hardware, a privacy request from a participant of the plurality of participants. The privacy request indicates a privacy condition associated with the participant in the speech environment. The method further includes segmenting, by the data processing hardware, the audio data into a plurality of segments. For each segment of the audio data, the method includes determining, by the data processing hardware, from among the plurality of participants, an identity of a speaker of a corresponding segment of the audio data based on the image data. For each segment of the audio data, the method also includes determining, by the data processing hardware, whether the identity of the speaker of the corresponding segment includes the participant associated with the privacy condition indicated by the received privacy request. When the identity of the speaker of the corresponding segment includes the participant, the method includes applying the privacy condition to the corresponding segment. The method further includes processing, by the data processing hardware, the plurality of segments of the audio data to determine a transcript for the audio data.

Implementations of the disclosure may include one or more of the following optional features. In some implementations, applying the privacy condition to the corresponding segment includes deleting the corresponding segment of the audio data after determining the transcript. Additionally or alternatively, applying the privacy condition to the corresponding segment may include augmenting a corresponding segment of the image data to visually conceal the identity of the speaker of the corresponding segment of the audio data.

In some examples, for each portion of the transcript that corresponds to one of the segments of the audio data applying the privacy condition, processing the plurality of segments of the audio data to determine the transcript for the audio data includes modifying the corresponding portion of the transcript to not include the identity of the speaker. Optionally, for each segment of the audio data applying the privacy condition, processing the plurality of segments of the audio data to determine the transcript for the audio data may include omitting transcribing the corresponding segment of the audio data. The privacy condition may include a content-specific condition, the content-specific condition indicating a type of content to exclude from the transcript.

In some configurations, determining, from among the plurality of participants, the identity of the speaker of the corresponding segment of the audio data includes determining a plurality of candidate identities for the speaker based on the image data. Here, for each candidate identity of the plurality of candidate identities, generating a confidence score indicating a likelihood that a face of a corresponding candidate identity based on the image data includes a speaking face of the corresponding segment of the audio data. In this configuration, the method includes selecting the identity of the speaker of the corresponding segment of the audio data as the candidate identity of the plurality of candidate identifies associated with the highest confidence score.

In some implementations, the data processing hardware resides on a device that is local to at least one participant of the plurality of participants. The image data may include high-definition video processed by the data processing hardware. Processing the plurality of segments of the audio data to determine a transcript for the audio data may include processing the image data to determine the transcript.

Another aspect of the disclosure provides a system for privacy-aware transcription. The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that when executed on the data processing hardware cause the data processing hardware to perform operations. The operations include receiving an audio-visual signal including audio data and image data. The audio data corresponds to speech utterances from a plurality of participants in a speech environment and the image data represents faces of the plurality of participants in the speech environment. The operations also include receiving a privacy request from a participant of the plurality of participants, the privacy request indicating a privacy condition associated with the participant in the speech environment. The method further includes segmenting the audio data into a plurality of segments. For each segment of the audio data, the operations include determining from among the plurality of participants, an identity of a speaker of a corresponding segment of the audio data based on the image data. For each segment of the audio data, the operations also include determining whether the identity of the speaker of the corresponding segment includes the participant associated with the privacy condition indicated by the received privacy request. When the identity of the speaker of the corresponding segment includes the participant, the operations include applying the privacy condition to the corresponding segment. The operations further include processing the plurality of segments of the audio data to determine a transcript for the audio data.

This aspect may include one or more of the following optional features. In some examples, applying the privacy condition to the corresponding segment includes deleting the corresponding segment of the audio data after determining the transcript. Optionally, applying the privacy condition to the corresponding segment may include augmenting a corresponding segment of the image data to visually conceal the identity of the speaker of the corresponding segment of the audio data.

In some configurations, processing the plurality of segments of the audio data to determine the transcript for the audio data includes, for each portion of the transcript that corresponds to one of the segments of the audio data applying the privacy condition, modifying the corresponding portion of the transcript to not include the identity of the speaker. Additionally or alternatively, processing the plurality of segments of the audio data to determine the transcript for the audio data may include, for each segment of the audio data applying the privacy condition, omitting transcribing the corresponding segment of the audio data. The privacy condition may include a content-specific condition, the content-specific condition indicating a type of content to exclude from the transcript.

In some implementations, the operation of determining, from among the plurality of participants, the identity of the speaker of the corresponding segment of the audio data includes determining a plurality of candidate identities for the speaker based on the image data. This implementation includes, for each candidate identity of the plurality of candidate identities, generating a confidence score indicating a likelihood that a face of a corresponding candidate identity based on the image data includes a speaking face of the corresponding segment of the audio data. This implementation also includes selecting the identity of the speaker of the corresponding segment of the audio data as the candidate identify of the plurality of candidate identities associated with the highest confidence score.

In some examples, the data processing hardware resides on a device that is local to at least one participant of the plurality of participants. The image data may include high-definition video processed by the data processing hardware. Processing the plurality of segments of the audio data to determine a transcript for the audio data may include processing the image data to determine the transcript.

The privacy of data used by, and generated by, video conferencing systems is an important aspect of such systems. Meeting participants may have their own individual view on privacy in relation to the audio and video data acquired during the meeting. There is, therefore, the technical problem of how to provide a video conferencing system that can accurately generate a transcript for a video conference meeting while also catering for such privacy requirements in a reliable and an accurate way. Implementations of the present disclosure provide a technical solution by enabling participants of the meeting to set their own privacy configuration (e.g. opt-in or opt-out of various features of the video conferencing system), with the video conferencing system then implementing the participant's desires accurately and effectively due to the video conferencing system, when generating a transcript, identifying spoken contributions from participants based not only on audio captured during the meeting but also based on video captured during the meeting - this ensures greater accuracy of the identification of the contributors to the video conference, thereby enabling improved accuracy of the transcript while at the same time enabling the accurate and reliable implementation of the participants' bespoke privacy requirements. In other words, a more accurate, reliable and flexible video conferencing system is provided.

Moreover, in some implementations, the process of generating the transcript of the video conference is performed locally to one or more of the participants of the video conference, e.g. by a device in the same room as those participants. In other words, in such implementations, the process of generating the transcript is not performed remotely, such as via one or more remote/cloud servers. This helps ensure that certain privacy desires can be met while also ensuring that full/original resolution and full/original quality video data captured locally is available for use when identifying the speakers during the video conference (as opposed to remote servers operating on lower-resolution and/or lower-quality video, which may detract from accuracy of speaker identification).

In an assembly environment (also referred to generally as an environment), people gather together to communicate thoughts, ideas, schedules, or other concerns. The assembly environment serves as a shared space for its participants. This shared space may be a physical space, such as a meeting room or a classroom, a virtual space (e.g., a virtual meeting room), or any combination thereof. The environment may be a centralized location (e.g., hosted locally) or a decentralized location (e.g., hosted virtually). For example, the environment is a single room where participants gather, such as a conference room or a classroom. In some implementations, the environment is more than one shared space linked together to form the assembly of participants. For instance, a meeting has a host location (e.g., where a coordinator or a presenter of a meeting may be located) and also one or more remote locations attending the meeting (e.g., using real-time communication applications). In other words, a business hosts a meeting from an office in Chicago, but other offices of the business (e.g., in San Francisco or New York) remotely attend the meeting. For instance, there are many businesses who have large meetings across several offices where each office has a meeting space that participates in the meeting. This is especially true as it is becoming more common for members of a team to be distributed across a company (i.e., in more than one location) or even working remotely. Moreover, as applications have become more robust for real-time communication, environments may be hosted for remote offices, remote employees, remote partners (e.g., business partners), remote customers, etc. Therefore, environments have evolved to accommodate a wide array of assembly logistics.

Generally as a space for communication, the environment hosts multiple participants. Here, each participant may contribute audio content (e.g., audible utterances by speaking) and/or visual content (e.g., actions of a participant) while present in the environment. With more than one participant in the environment, there are benefits to track and/or to record the participation of any or all participants. This is especially true as environments accommodate a wide array of assembly logistics. For instance, when the Chicago office hosts a meeting with both the New York office and the San Francisco office remotely attending, it may be difficult for someone in the Chicago office to identify a speaker in one of the remote locations. To illustrate, the Chicago office may include a video feed that captures the meeting room of each office that is remote to the Chicago office. Even with a video feed, the participant in the Chicago office may not be able to distinguish all of the participants in the New York office. For example, a speaker in the New York office is in a location far from a camera associated with the video feed making it difficult for a participant in the Chicago office to identify who the speaker is in the New York office. This may also be difficult when a Chicago-based participant is not familiar with other participants within the meeting (e.g., cannot identify a speaker by his/her voice). When a speaker cannot be identified, it can be problematic because the identity of the speaker may be a critical component during the meeting. In other words, it may be important to identify a speaker (or source of content) to understand takeaways/deliverables or generally to understand who shared what content. For instance, if Sally in the New York office assumes an action item deliverable to Johnny in the Chicago office, but Johnny could not identify that Sally assumed the action item, Johnny may have trouble following up on the action item later. In another scenario, because Johnny could not identify that Sally assumed the action item, Johnny may have incorrectly identified that Tracy (e.g., also in the New York office) assumed the action item. The same may be true on a basic level of simply conversing among participants. If Sally spoke about a certain topic, but Johnny thought it was Tracy speaking, Johnny may cause confusion when he engages Tracy about that topic at a later point in the meeting.

Another issue may arise when a speaker discusses names, acronyms, and/or jargon that another participant is not familiar with and/or cannot completely understand. In other words, Johnny may be discussing an issue that arose with a carrier used during shipping. Pete may chime in to help with Johnny's issue saying, "oh, you will want to speak with Teddy in logistics about that. " If Johnny is not familiar with Teddy and/or the logistics team, Johnny may make a note to speak with Freddie instead of Teddy. This also may occur for acronyms or other jargon used in a given industry. For example, if the Chicago office was having a meeting with a Seattle company where the Chicago office hosts the meeting and the Seattle company attends the meeting remotely, participants in the Chicago office may use acronyms and/or jargon that the Seattle company is not familiar with. Without a record or a transcription of the content presented by the Chicago office, the Seattle company, unfortunately, may comprehend less of the meeting (e.g., leading to a poor meeting). Additionally or alternatively, a poor connection between locations or with a meeting hosting platform may also complicate matters for a participant when the participant is trying to understand content during a meeting.

To overcome these issues, a transcription device is present in the environment that generates transcripts for the content that occurs within the environment (e.g., in real-time). When generating the transcript, the device can identify speakers (i.e., participants who generate audio content) and/or associate content with participants also present within the environment. With a transcript of the content presented in the environment, the transcription device is capable of memorializing takeaways and/or deliverables and providing a record of who originates what content that is accessible for participants to reference. For instance, a participant may reference the transcript during the meeting (e.g., in real-time or substantially real-time) or at some later time after the meeting. In other words, Johnny can refer to a display of the transcript generated by the transcription device to recognize that Teddy (not Freddie) is who he needs to talk to in logistics and that he should follow up with Sally (not Tracy) on that action item.

Unfortunately, although a transcript may solve some issues encountered in the environment, it presents an issue with regard to privacy. Here, privacy refers to having a state of being free from observation on the transcript generated by the transcription device. Although there may be many different types of privacy, some examples include content privacy or identity privacy. Here, content privacy is content-based such that it is desired that certain sensitive content is not memorialized in writing or a human-readable format (e.g., confidential content). For example, part of a meeting may include audio content about another employee that is not present in the meeting (e.g., managers discuss a human resource issue that arose). In this example, participants in the meeting would prefer that this part of the meeting regarding the other employee is not transcribed or otherwise memorialized. This may also include not memorializing the audio content that includes the content about the other employee. Here, since a traditional transcription device indiscriminately transcribes content, the meeting would not be able to utilize the traditional transcription device, at least during that portion of the meeting.

Identity privacy refers to a privacy that seeks to maintain an anonymity of a source of the content. For instance, a transcript often includes a label within the transcript that identifies a source of the transcribed content. For instance, labeling a speaker of the transcribed content may be referred to as speaker diarization to answer both "who spoke what" and "who spoke when". When the identity of the source of content is sensitive or the source (e.g., participant) generating the content prefers to mask his/her identity for whatever reason (e.g., personal reasons), the source does not want the label to be associated with the transcribed content. Note that here, unlike content privacy, the source does not mind the content being revealed in the transcript, but does not want an identifier (e.g., the label) to associate the content with the source. Since a traditional transcription device lacks the ability to accommodate these privacy concerns, participants may opt not to use a transcription device even though the aforementioned benefits are surrendered. In order to maintain these benefits and/or preserve the privacy of the participants, the environment may include a privacy-aware transcription device referred to as a transcriptor. In additional examples, when a camera is capturing video of the speaker who wants to remain anonymous, the speaker may choose to not have their recorded image (e.g., face) memorialized. This may include distorting the video/image frames of the speaker's face and/or overlaying graphics that mask the identity of the speaker so that the other individuals in the meeting cannot visually identify the speaker. Additionally or alternatively, the audio of the speaker's voice may be distorted (e.g., by passing the audio through a vocoder) to mask the speaker's voice in a way that anonymizes the speaker.

In some implementations, by processing the privacy on-device during transcription, privacy concerns are further bolstered such that the transcript does not leave the confines of the assembly environment (e.g., a meeting room or a classroom) providing the shared space for its participants. In other words, by using the transcriptor to generate a transcript on-device, speaker labels identifying speakers that want to remain anonymous can be removed on-device to alleviate any concerns that these speakers' identities will be exposed/compromised if processing of the transcript were to occur on a remote system (e.g., cloud environment). Stated different, there is not an un-redacted transcript generated by the transcriptor that may be shared or stored that jeopardizes the privacy of a participant.

Another technical affect of performing audio-video transcription (e.g., audio-video automated speech recognition (AVASR)) on device is a reduced bandwidth requirement since audio and image data (also referred to as video data) may be retained on locally on-device without the need to transmit it to a remote cloud server. For instance, if the video data were to be transmitted to the cloud, it is likely that it would first need to be compressed for transmission. Therefore, another technical effect of performing the video matching on the user device itself is that the video data matching may be performed using the uncompressed (highest quality) video data. The use of uncompressed video data makes it easier to recognize matches between audio data and faces of speakers so that speaker labels assigned to transcribed portions of the audio data spoken by speakers who do not want to be identified can be anonymized. At the same token, the video data capturing individuals' faces who do not want to be identified can be augmented/distorted/blurred to mask these individuals so that they cannot be visually identified if the video recording is shared. Similarly, the audio data representing utterances spoken by these individuals may be distorted to anonymous the speaking voices of these individuals who do not want to be identifiable. Referring to <FIG>, the environment <NUM> includes a plurality of participants <NUM>, 10a-j. Here, the environment <NUM> is a host meeting room with six participants 10a-f attending a meeting (e.g., a video conference) in the host meeting room. The environment <NUM> includes a display device <NUM> that receives a content feed <NUM> (also referred to as a multi-media feed, a content stream, or a feed) via a network <NUM> from a remote system <NUM>. The content feed <NUM> may be an audio feed <NUM> (i.e., audio data <NUM> such as audio content, an audio signal, or audio stream), a visual feed <NUM> (i.e., image data <NUM> such as video content, a video signal, or video stream), or some combination of both (e.g., also referred to as an audio-visual feed, an audio-visual signal, or an audio-visual stream). The display device <NUM> includes, or communicates with, a display <NUM> capable of displaying video content <NUM> and a speaker for audible output of the audio content <NUM>. Some examples of a display device <NUM> include a computer, a laptop, a mobile computing device, a television, a monitor, a smart device (e.g., smart speaker, smart display, smart appliance), a wearable device, etc. In some examples, the display device <NUM> includes audio-visual feeds <NUM> of other meeting rooms attending the meeting. For example, <FIG> depict two feeds <NUM>, 112a-b where each feed <NUM> corresponds to a different remote meeting room. Here, the first feed 112a includes three participants <NUM>, <NUM>-i while the second feed 112b includes a single participant <NUM>, 10j (e.g., an employee working remotely from a home office). To continue the previous example, the first feed 112a may correspond to a feed <NUM> from the New York office, the second feed 112b corresponds to a feed <NUM> from the San Francisco office, and the host meeting room <NUM> corresponds to the Chicago office.

The remote system <NUM> may be a distributed system (e.g., cloud computing environment or storage abstraction) having scalable/elastic resources <NUM>. The resources <NUM> include computing resources <NUM> (e.g., data processing hardware) and/or storage resources <NUM> (e.g. memory hardware). In some implementations, the remote system <NUM> hosts software that coordinates the environment <NUM> (e.g., on the computing resources <NUM>). For instance, the computing resources <NUM> of the remote system <NUM> execute software, such as a real-time communication application or a specialty meeting platform.

With continued reference to <FIG>, the environment <NUM> also includes a transcriptor <NUM>. The transcriptor <NUM> is configured to generate a transcript <NUM> for the content that occurs within the environment <NUM>. This content may be from a location where the transcriptor <NUM> is present (e.g., participants <NUM> in a meeting room <NUM> with the transcriptor <NUM>) and/or from a content feed <NUM> that communicates content to the location of the transcriptor <NUM>. In some examples, the display device <NUM> communicates one or more content feeds <NUM> to the transcriptor <NUM>. For example, the display device <NUM> includes speakers that output the audio content <NUM> of the content feed <NUM> to the transcriptor <NUM>. In some implementations, the transcriptor <NUM> is configured to receive the same content feed <NUM> as the display device <NUM>. In other words, the display device <NUM> may function as an extension of the transcriptor <NUM> by receiving the audio and video feeds of the content feed <NUM>. For instance, the display device <NUM> may include hardware <NUM> such data processing hardware <NUM> and memory hardware <NUM> in communication with the data processing hardware <NUM> that cause the data processing hardware <NUM> to execute the transcriptor <NUM>. In this relationship, the transcriptor <NUM> may receive the content feed <NUM> (e.g., audio and visual content/signals <NUM>, <NUM>) through a network connection rather than only audibly capturing the audio content/signal <NUM> relayed through a peripheral of the display device <NUM>, such as speakers. In some examples, this connectivity between the transcriptor <NUM> and the display device <NUM> enables the transcriptor <NUM> to seamlessly display the transcript <NUM> on the display/screen <NUM> of the display device <NUM> locally within the environment <NUM> (e.g., the host meeting room). In other configurations, the transcriptor <NUM> is located in the same local environment <NUM> as the display device <NUM>, but corresponds to a computing device separate from the display device <NUM>. In these configurations, the transcriptor <NUM> communicates with the display device <NUM> via a wired or wireless connection. For instance, the transcriptor <NUM> has one or more ports that permit a wired/wireless connection such that the display device <NUM> functions a peripheral to the transcriptor <NUM>. Additionally or alternatively, an application that forms the environment <NUM> may be compatible with the transcriptor <NUM>. For instance, the transcriptor <NUM> is configured as an input/output (I/O) device within the application such that audio and/or visual signals coordinated by the application are channeled to the transcriptor <NUM> (e.g., in addition to the display device <NUM>).

In some examples, the transcriptor <NUM> (and optionally the display device <NUM>) is portable such that the transcriptor <NUM> may be transferred between meeting rooms. In some implementations, the transcriptor <NUM> is configured with processing capabilities (e.g., processing hardware/software) to process the audio and video content <NUM> and generate the transcript <NUM> when the content <NUM> is being presented in the environment <NUM>. In other words, the transcriptor <NUM> is configured to process content <NUM> (e.g., audio and/or visual content <NUM>, <NUM>) locally at the transcriptor <NUM> to generate the transcript <NUM> without any additional remote processing (e.g., at the remote system <NUM>). Here, this type of processing is referred to as on-device processing. Unlike remote processing that often uses low fidelity, compressed video on server-based applications due to bandwidth constraints, on-device processing may be free of bandwidth constraints and thus allow the transcriptor <NUM> to utilize more accurate high definition video with high fidelity when processing visual content. Moreover, this on-device processing may allow an identity of a speaker to be tracked in real-time without delays due to latency that may occur if audio and/or visual signals <NUM>, <NUM> were, in some degree, remotely processed (e.g., in the remote computing system <NUM> connected to the transcriptor <NUM>). In order to process content at the transcriptor <NUM>, the transcriptor <NUM> includes the hardware <NUM> such as the data processing hardware <NUM> and the memory hardware <NUM> in communication with the data processing hardware <NUM>. Some examples of data processing hardware <NUM> include a central processing unit (CPU), a graphics processing unit (GPU), or a tensor processing unit (TPU).

In some implementations, the transcriptor <NUM> executes on the remote system <NUM> by receiving content <NUM> (audio and video data <NUM>, <NUM>) from each of the first and second feeds 112a-b, as well as a feed <NUM> from the meeting room environment <NUM>. For instance, the data processing hardware <NUM> of the remote system <NUM> may execute instructions stored on the memory hardware <NUM> of the remote system <NUM> for executing the transcriptor <NUM>. Here, the transcriptor <NUM> may process the audio data <NUM> and image data <NUM> to generate the transcript <NUM>. For example, the transcriptor <NUM> may generate the transcript <NUM> and transmit the transcript <NUM> over the network <NUM> to the display device <NUM> for display thereon. The transcriptor <NUM> may similarly transmit the transcript <NUM> to computing devices/ display devices associated with the participants <NUM>-i corresponding to the first feed and/or the participant 10j corresponding to the second feed 10j.

In addition to processing hardware <NUM>, the transcriptor <NUM> includes peripherals <NUM>. For instance, in order to process audio content, the transcriptor <NUM> includes an audio capturing device <NUM>, 216a (e.g., a microphone) that captures a sound (e.g., a speech utterance) about the transcriptor <NUM> and converts the sound into an audio signal <NUM> (<FIG> and <FIG>) (or audio data <NUM>). The audio signal <NUM> may then be used by the transcriptor <NUM> to generate the transcript <NUM>.

In some examples, the transcriptor <NUM> also includes an image capturing device <NUM>, 216b as a peripheral <NUM>. Here, the image capturing device 216b (e.g., one or more cameras) may capture image data <NUM> (<FIG> and <FIG>) as an additional source of input (e.g., a visual input) that, in combination with an audio signal <NUM>, aids in identifying which participant <NUM> in the multi-participant environment <NUM> is speaking (i.e., a speaker). In other words, by including both the audio capturing device 216a and the image capturing device 216b, a transcriptor <NUM> may increase its accuracy for speaker identification because the transcriptor <NUM> may process image data <NUM> captured by image capturing device 216b to identify visual features (e.g., facial features) that indicate which participant <NUM>, among the multiple participants 10a-10j, is speaking (i.e., generating utterances <NUM>) at a particular instance. In some configurations, the image capturing device 216b is configured to capture <NUM>-degrees about the transcriptor <NUM> to capture a full view of the environment <NUM>. For instance, the image capturing device 216b includes an array of cameras configured to capture the <NUM>-degree view.

Additionally or alternatively, using image data <NUM> may improve the transcript <NUM> when a participant <NUM> has a speech disability. For instance, the transcriptor <NUM> may have difficulty generating a transcript for a speaker with a speech disability that causes the speaker to have issues articulating speech. To overcome inaccuracies in the transcript <NUM> caused by such articulation issues, the transcriptor <NUM> (e.g., at an automatic speech recognition (ASR) module <NUM> of <FIG> and <FIG>) may be made aware of the articulation issue during generation of the transcript <NUM>. By being aware of the issue, the transcriptor <NUM> may accommodate for the issue by leveraging image data <NUM> representing the face of the participant <NUM> while speaking to generate an improved, or otherwise more accurate, transcript <NUM> than if the transcript <NUM> were based on the audio data <NUM> of the participant <NUM> alone. Here, certain speech disabilities may be noticeable in the image data <NUM> from the image capturing device 216b. For example, in the case of speech dysarthria, neural muscular disorders causing lip motion affecting articulation may be recognizable in the images <NUM>. Moreover, techniques may be employed where image data <NUM> can be analyzed to correlate the lip motion of participants <NUM> with particular speech disorders into the speech intended by these participants <NUM> to thereby improve automatic speech recognition in ways that would not be possible using audio data <NUM> alone. In some implementations, by using the images <NUM> as an input to the transcriptor <NUM>, the transcriptor <NUM> identifies a potential articulation issue and factors this issue to improve generation of the transcription <NUM> during ASR.

In some implementations, such as <FIG>, the transcriptor <NUM> is privacy-aware such that a participant <NUM> may opt out of having any of his or her speech and/or image information shared (e.g., in a transcript <NUM> or visual feed <NUM>, <NUM>). Here, one or more participants <NUM> communicate a privacy request <NUM> that indicates a privacy condition for a participant <NUM> during participation in the video conference environment <NUM>. In some examples, the privacy request <NUM> corresponds to configuration settings for the transcriptor <NUM>. The privacy request <NUM> may occur before, during, or at the onset of a meeting or communication session with the transcriptor <NUM>. In some configurations, the transcriptor <NUM> includes a profile (e.g., profile <NUM> as shown in <FIG>) that indicates one or more privacy requests <NUM> for the participant <NUM> (e.g., the individual profiles <NUM>, 510a-n of <FIG>). Here, the profile <NUM> may be stored on-device (e.g., in the memory hardware <NUM>) or stored off-device (e.g., in the remote storage resources <NUM>) and accessed by the transcriptor <NUM>. The profile <NUM> may be configured before the communication session and may include an image (e.g., image data <NUM>) of the respective participant's <NUM> face so the participant <NUM> can be correlated with respective portions of the received video content <NUM>. That is, an individual profile <NUM> for a respective participant <NUM> may be accessed when video content <NUM> of that participant <NUM> in the content feed <NUM> matches the facial image associated with the individual profile <NUM>. With an individual profile <NUM>, the participant's privacy settings may be applied during each communication session in which the participant <NUM> participates. In these examples, the transcriptor <NUM> may recognize the participant <NUM> (e.g., based on image data <NUM> received at the transcriptor <NUM>) and apply the appropriate settings for the participant <NUM>. For instance, the profile <NUM> may include an individual profile <NUM>, 510b for a particular participant <NUM>, 10b that indicates the particular participant 10b doesn't mind being seen (i.e., included in a visual feed <NUM>), but does not want to be heard (i.e., not included in audio feed <NUM>) nor have his/her utterances <NUM> transcribed (i.e., not included speech in the transcript <NUM>), while another individual profile <NUM>, 510c for another participant <NUM>, 10c may not want to be seen (i.e., not included in the visual feed <NUM>), but is fine with having his/her utterances recorded and/or transcribed (i.e., included in the audio feed <NUM> and included in the transcript <NUM>).

Referring to <FIG>, the third participant 10c has submitted a privacy request <NUM> with a privacy condition that indicates that the third participant 10c does not mind being seen or heard, but does not want the transcript <NUM> to include an identifier <NUM> (e.g., a label of the identity for the speaker) for the third participant 10c when the third participant 10c speaks (i.e., a privacy request <NUM> for identity privacy). In other words, the third participant 10c does not want his or her identity shared or stored; therefore, the third participant 10c opts for the transcript <NUM> to not include an identifier <NUM> associated with the third participant 10c that reveals his or her identity. Here, although <FIG> illustrates the transcript <NUM> with a redacted gray box where the identifier <NUM> of the Speaker <NUM> would exist, the transcriptor <NUM> may also remove the identifier <NUM> completely or obscure the identifier <NUM> in other ways that prevent the identity of the speaker associated with the privacy request <NUM> from being revealed by the transcriptor <NUM>. In other words, <FIG> illustrates that the transcriptor <NUM> modifies a portion of the transcript <NUM> to not include the identity of the speaker (e.g., by removing or obscuring the identifier <NUM>).

<FIG> is similar to <FIG> except that the third participant 10c who communicates the privacy request <NUM> requests to not be seen in any visual feed <NUM>, <NUM> of the environment <NUM> (e.g., another form of identity privacy). Here, the requesting participant 10c may not mind being heard, but prefers to visually conceal his or her visual identity (i.e., not share nor store his or her visual identity in a visual feed <NUM>, <NUM>). In this circumstance, the transcriptor <NUM> is configured to blur, distort, or otherwise obscure the visual presence of the requesting participant 10c throughout a communication session between the participants <NUM>, 10a-10j. For instance, the transcriptor <NUM> determines, from image data <NUM> received from one or more content feeds <NUM>, the location of the requestor 10c at any particular instance and applies an abstraction <NUM> to any physical features of the requestor (e.g., a blurring) that are communicated through the transcriptor <NUM>. That is, when the image data <NUM> is displayed on the screen <NUM> of the display device <NUM>, as well as displayed on screens in the remote environments associated with the participants <NUM>-10j, the abstraction <NUM> is overlaid over at least the face of the requestor 10c so that the requestor 10c cannot be visually identified. In some examples, the individual profile <NUM> for a participant <NUM> identifies whether the participant <NUM> wants to be blurred or obscured (i.e., distorted) or removed completely (e.g., as shown in <FIG>). Accordingly, the transcriptor <NUM> is configured to augment, modify, or remove portions of video data <NUM> to conceal a participant's visual identity.

In contrast, <FIG> illustrates an example where a privacy request <NUM> from the third participant 10c requests that the transcriptor <NUM> not track either a visual representation of the third participant 10c or speech information of the third participant 10c. As used herein, 'speech information' refers to audio data <NUM> corresponding to utterances <NUM> spoken by the participant 10c as well as transcripts <NUM> recognized from the audio data <NUM> corresponding to the utterances <NUM> spoken by the participant 10c. In this example, the participant 10c may be heard during the meeting, but the transcriptor <NUM> does not memorialize the participant 10c audibly or visually (e.g., by video feed <NUM> or in a transcript <NUM>). This approach may preserve the participant's 10c privacy by having no record of any speech information of the participant 10c in the transcript <NUM> or any identifier <NUM> identifying the participant 10c in the transcript <NUM>. For instance, the transcriptor <NUM> may omit portions of text in the transcript <NUM> altogether that transcribe utterances <NUM> spoken by the participant 10c, or the transcript <NUM> may leave these portions of the text in but not apply the identifier <NUM> identifying the participant 10c. The transcriptor <NUM> may, however, apply some other arbitrary identifier that does not personally identify the participant 10c, but merely delineates these portions of the text in the transcription <NUM> from other portions corresponding to utterances <NUM> spoken by the other participants 10a, 10b, 10d-10j. In other words, a participant <NUM> may request (e.g., via the privacy request <NUM>) that the transcript <NUM> and any other record generated by the transcriptor <NUM> have no record of the participant's participation in the communication session.

In contrast to the identity privacy requests <NUM>, <FIG> depicts a content privacy request <NUM>. In this example, the third participant 10c communicates a privacy request <NUM> that the transcriptor <NUM> does not include any content from the third participant 10c in the transcript <NUM>. Here, the third participant 10c makes such a privacy request <NUM> because the third participant 10c is going to discuss sensitive content (e.g., confidential information) during the meeting. Due to the sensitive nature of the content, the third participant 10c takes the precaution that the transcriptor <NUM> does not memorialize the audio content <NUM> associated with the third participant 10c in the transcript <NUM>. In some implementations, the transcriptor <NUM> is configured to receive a privacy request <NUM> that identifies a type of content that one or more participants <NUM> do not want included in the transcript <NUM> (e.g., by keywords) and to determine when that type of content occurs during the communication session in order to exclude it from the transcript <NUM>. In these implementations, not all audio content <NUM> from a particular participant <NUM> is excluded from the transcript <NUM>, only content-specific audio such that the particular participant still may discuss other types of content and be included in the transcript <NUM>. For instance, the third participant 10c communicates a privacy request <NUM> that requests that the transcriptor <NUM> does not transcribe audio content about Mike. In this instance, when the third participant 10c discusses Mike, the transcriptor <NUM> does not transcribe this audio content <NUM>, but when the third participant talks about other topics (e.g., the weather), the transcriptor <NUM> does transcribe this audio content <NUM>. The participant 10c may similarly set time boundaries to that the transcriptor <NUM> does not memorialize any audio content <NUM> for a length of time, e.g., the next <NUM>-minutes.

<FIG> and <FIG> are examples of the transcriptor <NUM>. The transcriptor <NUM> generally includes a diarization module <NUM> and an ASR module <NUM> (e.g., an AVASR module). The diarization module <NUM> is configured to receive audio data <NUM> that corresponds to the utterances <NUM> from the participants <NUM> of the communication session (e.g., captured by the audio capturing device 216a) and image data <NUM> representing faces of the participants <NUM> of the communication session, segment the audio data <NUM> into a plurality of segments <NUM>, 222a-n (e.g., fixed-length segments or variable-length segments), and generate diarization results <NUM> that include a corresponding speaker label <NUM> assigned to each segment <NUM> using a probability model (e.g., a probabilistic generative model) based on the audio data <NUM> and the image data <NUM>. In other words, the diarization module <NUM> includes a series of speaker recognition tasks with short utterances (e.g., segments <NUM>) and determines whether two segments <NUM> of a given conversation were spoken by the same participant <NUM>. Simultaneously, the diarization module <NUM> may execute a face tracking routine to identify which participant <NUM> is speaking during which segment <NUM> to further optimize speaker recognition. The diarization module <NUM> is then configured to repeat the process for all segments <NUM> of the conversation. Here, the diarization results <NUM> provide time-stamped speaker labels <NUM>, 226a-e for the received audio data <NUM> that not only identify who is speaking during a given segment <NUM>, but also identify when speaker changes occur between adjacent segments <NUM>. Here, the speaker labels <NUM> may serve as identifiers <NUM> within the transcript <NUM>.

In some examples, the transcriptor <NUM> receives the privacy request <NUM> at the diarization module <NUM>. Since the diarization module <NUM> identifies speaker labels <NUM> or identifiers <NUM>, the diarization module <NUM> may advantageously resolve a privacy request <NUM> that corresponds to an identity-based privacy request <NUM>. In other words, the diarization module <NUM> receives a privacy request <NUM> when the privacy request <NUM> requests not to identify a participant <NUM> by an identifier <NUM> such as the label <NUM> when the participant <NUM> is the speaker. When the diarization module <NUM> receives a privacy request <NUM>, the diarization module <NUM> is configured to determine whether the participant <NUM> corresponding to the request <NUM> matches the label <NUM> generated for a given segment <NUM>. In some examples, an image of the participant's <NUM> face may be used to associate the participant <NUM> with the label <NUM> for that participant <NUM>. When the label <NUM> for a segment <NUM> matches an identity of the participant <NUM> corresponding to the request <NUM>, the diarization module <NUM> may prevent the transcriptor <NUM> from applying the label <NUM> or identifier <NUM> to a corresponding portion of the resulting transcription <NUM> that transcribes the particular segment <NUM> into text. When the label <NUM> for a segment <NUM> fails to match an identity of the participant <NUM> corresponding to the request <NUM>, the diarization module <NUM> may allow the transcriptor to apply the label <NUM> and identifier <NUM> to the portion of the resulting transcription <NUM> that transcribes the particular segment into text. In some implementations, when the diarization module <NUM> receives the request <NUM>, the ASR module <NUM> is configured to wait to transcribe the audio data <NUM> from utterances <NUM>. In other implementations, the ASR module <NUM> transcribes in real-time and the resulting transcription <NUM> removes labels <NUM> and identifiers <NUM> in real-time for any participants <NUM> providing privacy requests <NUM> opting out from having their speech information transcribed. Optionally, the diarization module <NUM> may further distort the audio data <NUM> associated with these participants <NUM> seeking privacy so that their speaking voice is altered in a way that cannot be used to identify the participant <NUM>.

The ASR module <NUM> is configured to receive the audio data <NUM> corresponding to the utterances <NUM> and the image data <NUM> representing faces of the participants <NUM> while speaking the utterances <NUM>. Using the image data <NUM>, the ASR module <NUM> transcribes the audio data <NUM> into corresponding ASR results <NUM>. Here, the ASR result <NUM> refers to a textual transcription of the audio data <NUM> (e.g., the transcript <NUM>). In some examples, the ASR module <NUM> communicates with the diarization module <NUM> to utilize the diarization results <NUM> associated with the audio data <NUM> for improving speech recognition based on utterances <NUM>. For instance, the ASR module <NUM> may apply different speech recognition models (e.g., language models, prosody models) for different speakers identified from the diarization results <NUM>. Additionally or alternatively, the ASR module <NUM> and/or the diarization module <NUM> (or some other component of the transcriptor <NUM>) may index a transcription <NUM> of the audio data <NUM> using the time-stamped speaker labels <NUM> predicted for each segment <NUM> obtained from the diarization results <NUM>. Stated differently, the ASR module <NUM> uses the speaker labels <NUM> from the diarization module <NUM> to generate the identifiers <NUM> for speakers within the transcript <NUM>. As shown in <FIG>, the transcript <NUM> for a communication session within the environment <NUM> may be indexed by speaker/participant <NUM> to associate portions of the transcript <NUM> with the respective speaker/participant <NUM> in order to identify what each speaker/participant <NUM> said.

In some configurations, the ASR module <NUM> receives the privacy request <NUM> for the transcriptor <NUM>. For example, the ASR module <NUM> receives the privacy request <NUM> for the transcriptor <NUM> whenever the privacy request <NUM> corresponds to a request <NUM> not to transcribe speech for a particular participant <NUM>. In other words, the ASR module <NUM> may receive the privacy request <NUM> whenever the request <NUM> is not a label/identifier based privacy request <NUM>. In some examples, when the ASR module <NUM> receives a privacy request <NUM>, the ASR module <NUM> first identifies the participant <NUM> corresponding to the privacy request <NUM> based on the speaker labels <NUM> determined by the diarization module <NUM>. Then, when the ASR module <NUM> encounters speech to be transcribed for that participant <NUM>, the ARS module <NUM> applies the privacy request <NUM>. For instance, when the privacy request <NUM> requests not to transcribe speech for that particular participant <NUM>, the ASR module <NUM> does not transcribe any speech for that participant and waits for speech to occur by a different participant <NUM>.

Referring to <FIG>, in some implementations, the transcriptor <NUM> includes a detector <NUM> for executing the face tracking routine. In these implementations, the transcriptor <NUM> first processes the audio data <NUM> to generate one or more candidate identities for the speaker. For example, for each segment <NUM>, the diarization module <NUM> may include multiple labels <NUM>, 226a<NUM>-<NUM> as candidate identities for the speaker. In other words, the model may be a probability model that outputs multiple labels <NUM>, 226a<NUM>-<NUM> for each segment <NUM> where each label <NUM> of the multiple labels <NUM>, 226a<NUM>-<NUM> is a potential candidate that identifies the speaker. Here, the detector <NUM> of the transcriptor <NUM> uses the images <NUM>, 217a-n captured by the image capturing device 216b to determine which candidate identity had the best visual features indicating that he or she is the speaker of a particular segment <NUM>. In some configurations, the detector <NUM> generates a score <NUM> for each candidate identity where the score <NUM> indicates a confidence level that the candidate identity is the speaker based on the association between the audio signal (e.g., audio data <NUM>) and the visual signal (e.g., the captured images 217a-n). Here, the highest score <NUM> may indicate the greatest likelihood that the candidate identity is the speaker. In <FIG>, the diarization module <NUM> generates three labels 226a<NUM>-<NUM> at a particular segment <NUM>. The detector <NUM> generates a score <NUM> for each of these labels <NUM> (e.g., shown as three scores <NUM><NUM>-<NUM>) based on images <NUM> from the time in the audio data <NUM> where the segment <NUM> occurs. Here, <FIG> indicates the highest score <NUM> by a bolded square around the third label 226a<NUM> associated with the third score <NUM><NUM>. When the transcriptor <NUM> includes the detector <NUM>, the best candidate identity may be communicated to the ASR module <NUM> to form the identifier <NUM> of the transcript <NUM>.

Additionally or alternatively, the process may be reversed such that the transcriptor <NUM> first processes the image data <NUM> to generate one or more candidate identities for the speaker based on the image data <NUM>. Then for each candidate identity, the detector <NUM> generates a confidence score <NUM> indicating a likelihood that a face of the corresponding candidate identity includes a speaking face for a corresponding segment <NUM> of audio data <NUM>. For instance, the confidence score <NUM> for each candidate identity indicates a likelihood that a face of the corresponding candidate identity includes a speaking face during the image data <NUM> corresponding to an instance of time for a segment <NUM> of the audio data <NUM>. In other words, for each segment <NUM>, the detector <NUM> may score <NUM> whether image data <NUM> corresponding to participants <NUM> has facial expressions similar to or matching the facial expressions of a speaking face. Here, the detector <NUM> selects the identity of the speaker of the corresponding segment of the audio data <NUM> with the highest confidence score <NUM> as the candidate identity.

In some examples, the detector <NUM> is part of the ASR module <NUM>. Here, the ASR module <NUM> executes the face tracking routine by implementing an encoder frontend having an attention layer configured to receive a plurality of video tracks 217a-n of the image data <NUM>, whereby each video track is associated with a face of a respective participant. In these examples, the attention layer at the ASR module <NUM> is configured to determine a confidence score indicating a likelihood that the face of the respective person associated with the video face track includes a speaking face of the audio track. Additional concepts and features related to an audio-visual ASR module including an encoder front end having an attention layer for multi-speaker ASR recognition can be found in <CIT>, which is hereby incorporated by reference in its entirety.

In some configurations, the transcriptor <NUM> (e.g., at the ASR module <NUM>) is configured to support a multi-lingual environment <NUM>. For example, when the transcriptor <NUM> generates the transcript <NUM>, the transcriptor <NUM> is capable of generating the transcript <NUM> in different languages. This feature may enable the environment <NUM> to include a remote location that has one or more participants <NUM> that speak a different language than the host location. Moreover, in some situations, the speaker in a meeting may be a non-native speaker or a speaker where the language of the meeting is not the first language of the speaker. Here, a transcript <NUM> of the content from the speaker may assist other participants <NUM> in the meeting to understand the presented content. Additionally or alternatively, the transcriptor <NUM> may be used to provide a speaker with feedback on his or her pronunciation. Here, by combining the video and/or the audio data, the transcriptor <NUM> may indicate an incorrect pronunciation (e.g., allowing a speaker to learn and/or to adapt with the help of the transcriptor <NUM>). As such, the transcriptor <NUM> may provide a notification to the speaker that provides the feedback on his/her pronunciation.

<FIG> is an example arrangement of operations for a method <NUM> of transcribing content (e.g., at the data processing hardware <NUM> of the transcriptor <NUM>). At operation <NUM>, the method <NUM> includes receiving an audio-visual signal <NUM>, <NUM> including audio data <NUM> and image data <NUM>. The audio data <NUM> corresponds to speech utterances <NUM> from a plurality of participants <NUM>, 10a-n in a speech environment <NUM> and the image data <NUM> represents faces of the plurality of participants <NUM> in the speech environment <NUM>. At operation <NUM>, the method <NUM> includes receiving a privacy request <NUM> from a participant <NUM> of the plurality of participants 10a-n. The privacy request <NUM> indicates a privacy condition associated with the participant <NUM> in the speech environment <NUM>. At operation <NUM>, the method <NUM> segments the audio data <NUM> into a plurality of segments <NUM>, 222a-n. At operation <NUM>, the method <NUM> includes performing operations <NUM>, 308a-c for each segment <NUM> of the audio data <NUM>. At operation 308a, for each segment <NUM> of the audio data <NUM>, the method <NUM> includes determining from among the plurality of participants 10a-n, an identity of a speaker of a corresponding segment <NUM> of the audio data <NUM> based on the image data <NUM>. At operation 308b, for each segment <NUM> of the audio data <NUM>, the method <NUM> includes determining whether the identity of the speaker of the corresponding segment <NUM> includes the participant <NUM> associated with the privacy condition indicated by the received privacy request <NUM>. At operation 308c, for each segment <NUM> of the audio data <NUM>, when the identity of the speaker of the corresponding segment <NUM> includes the participant <NUM>, the method <NUM> includes applying the privacy condition to the corresponding segment <NUM>. At operation <NUM>, the method <NUM> includes processing the plurality of segments 222a-n of the audio data <NUM> to determine a transcript <NUM> for the audio data <NUM>.

For example, a user is provided with control over whether programs or features collect user information about that particular user or other users relevant to the program or feature. Each user for which personal information is to be collected is presented with one or more options to allow control over the information collection relevant to that user, to provide permission or authorization as to whether the information is collected and as to which portions of the information are to be collected. For example, users can be provided with one or more such control options over a communication network. In addition, certain data may be treated in one or more ways before it is stored or used so that personally identifiable information is removed. As one example, a user's identity may be treated so that no personally identifiable information can be determined.

The computing device <NUM> includes a processor <NUM> (e.g., data processing hardware), memory <NUM> (e.g., memory hardware), a storage device <NUM>, a high-speed interface/controller <NUM> connecting to the memory <NUM> and high-speed expansion ports <NUM>, and a low speed interface/controller <NUM> connecting to a low speed bus <NUM> and a storage device <NUM>. Also, multiple computing devices <NUM> may be connected, with each device providing portions of the necessary operations (e.g., as a server bank, a group of blade servers, or a multiprocessor system).

For example, it may be implemented as a standard server 400a or multiple times in a group of such servers 400a, as a laptop computer 400b, or as part of a rack server system 400c.

Claim 1:
A method comprising:
receiving (<NUM>), at data processing hardware (<NUM>), an audio-visual signal (<NUM>, <NUM>) comprising audio data (<NUM>) and image data (<NUM>), the audio data (<NUM>) corresponding to speech utterances (<NUM>) from a plurality of participants (<NUM>) in a speech environment (<NUM>) and the image data (<NUM>) representing faces of the plurality of participants (<NUM>) in the speech environment (<NUM>);
receiving (<NUM>), at the data processing hardware (<NUM>), a privacy request (<NUM>) from a participant of the plurality of participants (<NUM>), the privacy request (<NUM>) indicating a privacy condition associated with the participant (<NUM>) in the speech environment (<NUM>);
segmenting (<NUM>), by the data processing hardware (<NUM>), the audio data (<NUM>) into a plurality of segments (<NUM>);
for each segment (<NUM>) of the audio data (<NUM>):
determining (308a), by the data processing hardware (<NUM>), from among the plurality of participants (<NUM>), an identity of a speaker of a corresponding segment (<NUM>) of the audio data based (<NUM>) on the image data (<NUM>);
determining (308b), by the data processing hardware (<NUM>), whether the identity of the speaker of the corresponding segment (<NUM>) comprises the participant (<NUM>) associated with the privacy condition indicated by the received privacy request (<NUM>); and
when the identity of the speaker of the corresponding segment (<NUM>) comprises the participant (<NUM>), applying (308c) the privacy condition to the corresponding segment (<NUM>); and
processing (<NUM>), by the data processing hardware (<NUM>), the plurality of segments (<NUM>) of the audio data (<NUM>) to determine a transcript (<NUM>) for the audio data (<NUM>).