Patent ID: 12242648

Like reference symbols in the various drawings indicate like elements.

DETAILED DESCRIPTION

The privacy of data used by, and generated by, video conferencing systems is an important aspect of such systems. Meeting participants may have their own individual view on privacy in relation to the audio and video data acquired during the meeting. There is, therefore, the technical problem of how to provide a video conferencing system that can accurately generate a transcript for a video conference meeting while also catering for such privacy requirements in a reliable and an accurate way. Implementations of the present disclosure provide a technical solution by enabling participants of the meeting to set their own privacy configuration (e.g. opt-in or opt-out of various features of the video conferencing system), with the video conferencing system then implementing the participant's desires accurately and effectively due to the video conferencing system, when generating a transcript, identifying spoken contributions from participants based not only on audio captured during the meeting but also based on video captured during the meeting—this ensures greater accuracy of the identification of the contributors to the video conference, thereby enabling improved accuracy of the transcript while at the same time enabling the accurate and reliable implementation of the participants' bespoke privacy requirements. In other words, a more accurate, reliable and flexible video conferencing system is provided.

Moreover, in some implementations, the process of generating the transcript of the video conference is performed locally to one or more of the participants of the video conference, e.g. by a device in the same room as those participants. In other words, in such implementations, the process of generating the transcript is not performed remotely, such as via one or more remote/cloud servers. This helps ensure that certain privacy desires can be met while also ensuring that full/original resolution and full/original quality video data captured locally is available for use when identifying the speakers during the video conference (as opposed to remote servers operating on lower-resolution and/or lower-quality video, which may detract from accuracy of speaker identification).

In an assembly environment (also referred to generally as an environment), people gather together to communicate thoughts, ideas, schedules, or other concerns. The assembly environment serves as a shared space for its participants. This shared space may be a physical space, such as a meeting room or a classroom, a virtual space (e.g., a virtual meeting room), or any combination thereof. The environment may be a centralized location (e.g., hosted locally) or a decentralized location (e.g., hosted virtually). For example, the environment is a single room where participants gather, such as a conference room or a classroom. In some implementations, the environment is more than one shared space linked together to form the assembly of participants. For instance, a meeting has a host location (e.g., where a coordinator or a presenter of a meeting may be located) and also one or more remote locations attending the meeting (e.g., using real-time communication applications). In other words, a business hosts a meeting from an office in Chicago, but other offices of the business (e.g., in San Francisco or New York) remotely attend the meeting. For instance, there are many businesses who have large meetings across several offices where each office has a meeting space that participates in the meeting. This is especially true as it is becoming more common for members of a team to be distributed across a company (i.e., in more than one location) or even working remotely. Moreover, as applications have become more robust for real-time communication, environments may be hosted for remote offices, remote employees, remote partners (e.g., business partners), remote customers, etc. Therefore, environments have evolved to accommodate a wide array of assembly logistics.

Generally as a space for communication, the environment hosts multiple participants. Here, each participant may contribute audio content (e.g., audible utterances by speaking) and/or visual content (e.g., actions of a participant) while present in the environment. With more than one participant in the environment, there are benefits to track and/or to record the participation of any or all participants. This is especially true as environments accommodate a wide array of assembly logistics. For instance, when the Chicago office hosts a meeting with both the New York office and the San Francisco office remotely attending, it may be difficult for someone in the Chicago office to identify a speaker in one of the remote locations. To illustrate, the Chicago office may include a video feed that captures the meeting room of each office that is remote to the Chicago office. Even with a video feed, the participant in the Chicago office may not be able to distinguish all of the participants in the New York office. For example, a speaker in the New York office is in a location far from a camera associated with the video feed making it difficult for a participant in the Chicago office to identify who the speaker is in the New York office. This may also be difficult when a Chicago-based participant is not familiar with other participants within the meeting (e.g., cannot identify a speaker by his/her voice). When a speaker cannot be identified, it can be problematic because the identity of the speaker may be a critical component during the meeting. In other words, it may be important to identify a speaker (or source of content) to understand takeaways/deliverables or generally to understand who shared what content. For instance, if Sally in the New York office assumes an action item deliverable to Johnny in the Chicago office, but Johnny could not identify that Sally assumed the action item, Johnny may have trouble following up on the action item later. In another scenario, because Johnny could not identify that Sally assumed the action item, Johnny may have incorrectly identified that Tracy (e.g., also in the New York office) assumed the action item. The same may be true on a basic level of simply conversing among participants. If Sally spoke about a certain topic, but Johnny thought it was Tracy speaking, Johnny may cause confusion when he engages Tracy about that topic at a later point in the meeting.

Another issue may arise when a speaker discusses names, acronyms, and/or jargon that another participant is not familiar with and/or cannot completely understand. In other words, Johnny may be discussing an issue that arose with a carrier used during shipping. Pete may chime in to help with Johnny's issue saying, “oh, you will want to speak with Teddy in logistics about that.” If Johnny is not familiar with Teddy and/or the logistics team, Johnny may make a note to speak with Freddie instead of Teddy. This also may occur for acronyms or other jargon used in a given industry. For example, if the Chicago office was having a meeting with a Seattle company where the Chicago office hosts the meeting and the Seattle company attends the meeting remotely, participants in the Chicago office may use acronyms and/or jargon that the Seattle company is not familiar with. Without a record or a transcription of the content presented by the Chicago office, the Seattle company, unfortunately, may comprehend less of the meeting (e.g., leading to a poor meeting). Additionally or alternatively, a poor connection between locations or with a meeting hosting platform may also complicate matters for a participant when the participant is trying to understand content during a meeting.

To overcome these issues, a transcription device is present in the environment that generates transcripts for the content that occurs within the environment (e.g., in real-time). When generating the transcript, the device can identify speakers (i.e., participants who generate audio content) and/or associate content with participants also present within the environment. With a transcript of the content presented in the environment, the transcription device is capable of memorializing takeaways and/or deliverables and providing a record of who originates what content that is accessible for participants to reference. For instance, a participant may reference the transcript during the meeting (e.g., in real-time or substantially real-time) or at some later time after the meeting. In other words, Johnny can refer to a display of the transcript generated by the transcription device to recognize that Teddy (not Freddie) is who he needs to talk to in logistics and that he should follow up with Sally (not Tracy) on that action item.

Unfortunately, although a transcript may solve some issues encountered in the environment, it presents an issue with regard to privacy. Here, privacy refers to having a state of being free from observation on the transcript generated by the transcription device. Although there may be many different types of privacy, some examples include content privacy or identity privacy. Here, content privacy is content-based such that it is desired that certain sensitive content is not memorialized in writing or a human-readable format (e.g., confidential content). For example, part of a meeting may include audio content about another employee that is not present in the meeting (e.g., managers discuss a human resource issue that arose). In this example, participants in the meeting would prefer that this part of the meeting regarding the other employee is not transcribed or otherwise memorialized. This may also include not memorializing the audio content that includes the content about the other employee. Here, since a traditional transcription device indiscriminately transcribes content, the meeting would not be able to utilize the traditional transcription device, at least during that portion of the meeting.

Identity privacy refers to a privacy that seeks to maintain an anonymity of a source of the content. For instance, a transcript often includes a label within the transcript that identifies a source of the transcribed content. For instance, labeling a speaker of the transcribed content may be referred to as speaker diarization to answer both “who spoke what” and “who spoke when”. When the identity of the source of content is sensitive or the source (e.g., participant) generating the content prefers to mask his/her identity for whatever reason (e.g., personal reasons), the source does not want the label to be associated with the transcribed content. Note that here, unlike content privacy, the source does not mind the content being revealed in the transcript, but does not want an identifier (e.g., the label) to associate the content with the source. Since a traditional transcription device lacks the ability to accommodate these privacy concerns, participants may opt not to use a transcription device even though the aforementioned benefits are surrendered. In order to maintain these benefits and/or preserve the privacy of the participants, the environment may include a privacy-aware transcription device referred to as a transcriptor. In additional examples, when a camera is capturing video of the speaker who wants to remain anonymous, the speaker may choose to not have their recorded image (e.g., face) memorialized. This may include distorting the video/image frames of the speaker's face and/or overlaying graphics that mask the identity of the speaker so that the other individuals in the meeting cannot visually identify the speaker. Additionally or alternatively, the audio of the speaker's voice may be distorted (e.g., by passing the audio through a vocoder) to mask the speaker's voice in a way that anonymizes the speaker.

In some implementations, by processing the privacy on-device during transcription, privacy concerns are further bolstered such that the transcript does not leave the confines of the assembly environment (e.g., a meeting room or a classroom) providing the shared space for its participants. In other words, by using the transcriptor to generate a transcript on-device, speaker labels identifying speakers that want to remain anonymous can be removed on-device to alleviate any concerns that these speakers' identities will be exposed/compromised if processing of the transcript were to occur on a remote system (e.g., cloud environment). Stated different, there is not an un-redacted transcript generated by the transcriptor that may be shared or stored that jeopardizes the privacy of a participant.

Another technical affect of performing audio-video transcription (e.g., audio-video automated speech recognition (AVASR)) on device is a reduced bandwidth requirement since audio and image data (also referred to as video data) may be retained on locally on-device without the need to transmit it to a remote cloud server. For instance, if the video data were to be transmitted to the cloud, it is likely that it would first need to be compressed for transmission. Therefore, another technical effect of performing the video matching on the user device itself is that the video data matching may be performed using the uncompressed (highest quality) video data. The use of uncompressed video data makes it easier to recognize matches between audio data and faces of speakers so that speaker labels assigned to transcribed portions of the audio data spoken by speakers who do not want to be identified can be anonymized. At the same token, the video data capturing individuals' faces who do not want to be identified can be augmented/distorted/blurred to mask these individuals so that they cannot be visually identified if the video recording is shared. Similarly, the audio data representing utterances spoken by these individuals may be distorted to anonymous the speaking voices of these individuals who do not want to be identifiable. Referring toFIGS.1A-1E, the environment100includes a plurality of participants10,10a-j. Here, the environment100is a host meeting room with six participants10a—f attending a meeting (e.g., a video conference) in the host meeting room. The environment100includes a display device110that receives a content feed112(also referred to as a multi-media feed, a content stream, or a feed) via a network120from a remote system130. The content feed112may be an audio feed218(i.e., audio data218such as audio content, an audio signal, or audio stream), a visual feed217(i.e., image data217such as video content, a video signal, or video stream), or some combination of both (e.g., also referred to as an audio-visual feed, an audio-visual signal, or an audio-visual stream). The display device110includes, or communicates with, a display111capable of displaying video content217and a speaker for audible output of the audio content218. Some examples of a display device110include a computer, a laptop, a mobile computing device, a television, a monitor, a smart device (e.g., smart speaker, smart display, smart appliance), a wearable device, etc. In some examples, the display device110includes audio-visual feeds112of other meeting rooms attending the meeting. For example,FIGS.1A-1Edepict two feeds112,112a-bwhere each feed112corresponds to a different remote meeting room. Here, the first feed112aincludes three participants10,10g-iwhile the second feed112bincludes a single participant10,10j(e.g., an employee working remotely from a home office). To continue the previous example, the first feed112amay correspond to a feed112from the New York office, the second feed112bcorresponds to a feed112from the San Francisco office, and the host meeting room100corresponds to the Chicago office.

The remote system130may be a distributed system (e.g., cloud computing environment or storage abstraction) having scalable/elastic resources132. The resources132include computing resources134(e.g., data processing hardware) and/or storage resources136(e.g. memory hardware). In some implementations, the remote system130hosts software that coordinates the environment100(e.g., on the computing resources132). For instance, the computing resources132of the remote system130execute software, such as a real-time communication application or a specialty meeting platform.

With continued reference toFIGS.1A-1E, the environment100also includes a transcriptor200. The transcriptor200is configured to generate a transcript202for the content that occurs within the environment100. This content may be from a location where the transcriptor200is present (e.g., participants10in a meeting room100with the transcriptor200) and/or from a content feed112that communicates content to the location of the transcriptor200. In some examples, the display device110communicates one or more content feeds112to the transcriptor200. For example, the display device110includes speakers that output the audio content218of the content feed112to the transcriptor200. In some implementations, the transcriptor200is configured to receive the same content feed112as the display device110. In other words, the display device110may function as an extension of the transcriptor200by receiving the audio and video feeds of the content feed112. For instance, the display device110may include hardware210such data processing hardware212and memory hardware214in communication with the data processing hardware212that cause the data processing hardware212to execute the transcriptor200. In this relationship, the transcriptor200may receive the content feed112(e.g., audio and visual content/signals218,217) through a network connection rather than only audibly capturing the audio content/signal218relayed through a peripheral of the display device110, such as speakers. In some examples, this connectivity between the transcriptor200and the display device110enables the transcriptor200to seamlessly display the transcript202on the display/screen111of the display device110locally within the environment100(e.g., the host meeting room). In other configurations, the transcriptor200is located in the same local environment110as the display device110, but corresponds to a computing device separate from the display device110. In these configurations, the transcriptor200communicates with the display device110via a wired or wireless connection. For instance, the transcriptor200has one or more ports that permit a wired/wireless connection such that the display device110functions a peripheral to the transcriptor200. Additionally or alternatively, an application that forms the environment100may be compatible with the transcriptor200. For instance, the transcriptor200is configured as an input/output (I/O) device within the application such that audio and/or visual signals coordinated by the application are channeled to the transcriptor200(e.g., in addition to the display device110).

In some examples, the transcriptor200(and optionally the display device110) is portable such that the transcriptor200may be transferred between meeting rooms. In some implementations, the transcriptor200is configured with processing capabilities (e.g., processing hardware/software) to process the audio and video content112and generate the transcript202when the content112is being presented in the environment100. In other words, the transcriptor200is configured to process content112(e.g., audio and/or visual content218,217) locally at the transcriptor200to generate the transcript202without any additional remote processing (e.g., at the remote system130). Here, this type of processing is referred to as on-device processing. Unlike remote processing that often uses low fidelity, compressed video on server-based applications due to bandwidth constraints, on-device processing may be free of bandwidth constraints and thus allow the transcriptor200to utilize more accurate high definition video with high fidelity when processing visual content. Moreover, this on-device processing may allow an identity of a speaker to be tracked in real-time without delays due to latency that may occur if audio and/or visual signals218,217were, in some degree, remotely processed (e.g., in the remote computing system130connected to the transcriptor200). In order to process content at the transcriptor200, the transcriptor200includes the hardware210such as the data processing hardware212and the memory hardware214in communication with the data processing hardware212. Some examples of data processing hardware212include a central processing unit (CPU), a graphics processing unit (GPU), or a tensor processing unit (TPU).

In some implementations, the transcriptor200executes on the remote system130by receiving content112(audio and video data217,218) from each of the first and second feeds112a-b, as well as a feed112from the meeting room environment100. For instance, the data processing hardware134of the remote system130may execute instructions stored on the memory hardware136of the remote system130for executing the transcriptor200. Here, the transcriptor200may process the audio data218and image data217to generate the transcript202. For example, the transcriptor200may generate the transcript202and transmit the transcript202over the network120to the display device110for display thereon. The transcriptor200may similarly transmit the transcript202to computing devices/display devices associated with the participants10g-icorresponding to the first feed and/or the participant10jcorresponding to the second feed10j.

In addition to processing hardware210, the transcriptor200includes peripherals216. For instance, in order to process audio content, the transcriptor200includes an audio capturing device216,216a(e.g., a microphone) that captures a sound (e.g., a speech utterance) about the transcriptor200and converts the sound into an audio signal218(FIGS.2A and2B) (or audio data218). The audio signal218may then be used by the transcriptor200to generate the transcript202.

In some examples, the transcriptor200also includes an image capturing device216,216bas a peripheral216. Here, the image capturing device216b(e.g., one or more cameras) may capture image data217(FIGS.2A and2B) as an additional source of input (e.g., a visual input) that, in combination with an audio signal218, aids in identifying which participant10in the multi-participant environment100is speaking (i.e., a speaker). In other words, by including both the audio capturing device216aand the image capturing device216b, a transcriptor200may increase its accuracy for speaker identification because the transcriptor200may process image data217captured by image capturing device216bto identify visual features (e.g., facial features) that indicate which participant10, among the multiple participants10a-10j, is speaking (i.e., generating utterances12) at a particular instance. In some configurations, the image capturing device216bis configured to capture 360-degrees about the transcriptor200to capture a full view of the environment100. For instance, the image capturing device216bincludes an array of cameras configured to capture the 360-degree view.

Additionally or alternatively, using image data217may improve the transcript202when a participant10has a speech disability. For instance, the transcriptor200may have difficulty generating a transcript for a speaker with a speech disability that causes the speaker to have issues articulating speech. To overcome inaccuracies in the transcript202caused by such articulation issues, the transcriptor200(e.g., at an automatic speech recognition (ASR) module230ofFIGS.2A and2B) may be made aware of the articulation issue during generation of the transcript202. By being aware of the issue, the transcriptor200may accommodate for the issue by leveraging image data217representing the face of the participant10while speaking to generate an improved, or otherwise more accurate, transcript202than if the transcript202were based on the audio data218of the participant10alone. Here, certain speech disabilities may be noticeable in the image data217from the image capturing device216b. For example, in the case of speech dysarthria, neural muscular disorders causing lip motion affecting articulation may be recognizable in the images217. Moreover, techniques may be employed where image data217can be analyzed to correlate the lip motion of participants10with particular speech disorders into the speech intended by these participants10to thereby improve automatic speech recognition in ways that would not be possible using audio data218alone. In some implementations, by using the images217as an input to the transcriptor200, the transcriptor200identifies a potential articulation issue and factors this issue to improve generation of the transcription202during ASR.

In some implementations, such asFIGS.1B-1E, the transcriptor200is privacy-aware such that a participant10may opt out of having any of his or her speech and/or image information shared (e.g., in a transcript202or visual feed112,217). Here, one or more participants10communicate a privacy request14that indicates a privacy condition for a participant10during participation in the video conference environment100. In some examples, the privacy request14corresponds to configuration settings for the transcriptor200. The privacy request14may occur before, during, or at the onset of a meeting or communication session with the transcriptor200. In some configurations, the transcriptor200includes a profile (e.g., profile500as shown inFIG.5) that indicates one or more privacy requests14for the participant10(e.g., the individual profiles510,510a-nofFIG.5). Here, the profile500may be stored on-device (e.g., in the memory hardware214) or stored off-device (e.g., in the remote storage resources136) and accessed by the transcriptor200. The profile500may be configured before the communication session and may include an image (e.g., image data217) of the respective participant's10face so the participant10can be correlated with respective portions of the received video content217. That is, an individual profile510for a respective participant10may be accessed when video content217of that participant10in the content feed112matches the facial image associated with the individual profile510. With an individual profile510, the participant's privacy settings may be applied during each communication session in which the participant10participates. In these examples, the transcriptor200may recognize the participant10(e.g., based on image data217received at the transcriptor200) and apply the appropriate settings for the participant10. For instance, the profile500may include an individual profile510,510bfor a particular participant10,10bthat indicates the particular participant10bdoesn't mind being seen (i.e., included in a visual feed217), but does not want to be heard (i.e., not included in audio feed218) nor have his/her utterances12transcribed (i.e., not included speech in the transcript202), while another individual profile510,510cfor another participant10,10cmay not want to be seen (i.e., not included in the visual feed217), but is fine with having his/her utterances recorded and/or transcribed (i.e., included in the audio feed218and included in the transcript202).

Referring toFIG.1B, the third participant10chas submitted a privacy request14with a privacy condition that indicates that the third participant10cdoes not mind being seen or heard, but does not want the transcript202to include an identifier204(e.g., a label of the identity for the speaker) for the third participant10cwhen the third participant10cspeaks (i.e., a privacy request14for identity privacy). In other words, the third participant10cdoes not want his or her identity shared or stored; therefore, the third participant10copts for the transcript202to not include an identifier204associated with the third participant10cthat reveals his or her identity. Here, althoughFIG.1Billustrates the transcript202with a redacted gray box where the identifier204of the Speaker 3 would exist, the transcriptor200may also remove the identifier204completely or obscure the identifier204in other ways that prevent the identity of the speaker associated with the privacy request14from being revealed by the transcriptor200. In other words,FIG.1Billustrates that the transcriptor200modifies a portion of the transcript202to not include the identity of the speaker (e.g., by removing or obscuring the identifier204).

FIG.1Cis similar toFIG.1Bexcept that the third participant10cwho communicates the privacy request14requests to not be seen in any visual feed112,217of the environment100(e.g., another form of identity privacy). Here, the requesting participant10cmay not mind being heard, but prefers to visually conceal his or her visual identity (i.e., not share nor store his or her visual identity in a visual feed112,217). In this circumstance, the transcriptor200is configured to blur, distort, or otherwise obscure the visual presence of the requesting participant10cthroughout a communication session between the participants10,10a-10j. For instance, the transcriptor200determines, from image data217received from one or more content feeds112, the location of the requestor10cat any particular instance and applies an abstraction119to any physical features of the requestor (e.g., a blurring) that are communicated through the transcriptor200. That is, when the image data217is displayed on the screen111of the display device110, as well as displayed on screens in the remote environments associated with the participants10g-10j, the abstraction119is overlaid over at least the face of the requestor10cso that the requestor10ccannot be visually identified. In some examples, the individual profile510for a participant10identifies whether the participant10wants to be blurred or obscured (i.e., distorted) or removed completely (e.g., as shown inFIG.5). Accordingly, the transcriptor200is configured to augment, modify, or remove portions of video data217to conceal a participant's visual identity.

In contrast,FIG.1Dillustrates an example where a privacy request14from the third participant10crequests that the transcriptor200not track either a visual representation of the third participant10cor speech information of the third participant10c. As used herein, ‘speech information’ refers to audio data218corresponding to utterances12spoken by the participant10cas well as transcripts202recognized from the audio data218corresponding to the utterances12spoken by the participant10c. In this example, the participant10cmay be heard during the meeting, but the transcriptor200does not memorialize the participant10caudibly or visually (e.g., by video feed217or in a transcript202). This approach may preserve the participant's10cprivacy by having no record of any speech information of the participant10cin the transcript202or any identifier204identifying the participant10cin the transcript202. For instance, the transcriptor200may omit portions of text in the transcript202altogether that transcribe utterances12spoken by the participant10c, or the transcript202may leave these portions of the text in but not apply the identifier204identifying the participant10c. The transcriptor200may, however, apply some other arbitrary identifier that does not personally identify the participant10c, but merely delineates these portions of the text in the transcription202from other portions corresponding to utterances12spoken by the other participants10a,10b,10d-10j. In other words, a participant10may request (e.g., via the privacy request14) that the transcript202and any other record generated by the transcriptor200have no record of the participant's participation in the communication session.

In contrast to the identity privacy requests14,FIG.1Edepicts a content privacy request14. In this example, the third participant10ccommunicates a privacy request14that the transcriptor200does not include any content from the third participant10cin the transcript202. Here, the third participant10cmakes such a privacy request14because the third participant10cis going to discuss sensitive content (e.g., confidential information) during the meeting. Due to the sensitive nature of the content, the third participant10ctakes the precaution that the transcriptor200does not memorialize the audio content218associated with the third participant10cin the transcript202. In some implementations, the transcriptor200is configured to receive a privacy request14that identifies a type of content that one or more participants10do not want included in the transcript202(e.g., by keywords) and to determine when that type of content occurs during the communication session in order to exclude it from the transcript202. In these implementations, not all audio content218from a particular participant10is excluded from the transcript202, only content-specific audio such that the particular participant still may discuss other types of content and be included in the transcript202. For instance, the third participant10ccommunicates a privacy request14that requests that the transcriptor200does not transcribe audio content about Mike. In this instance, when the third participant10cdiscusses Mike, the transcriptor200does not transcribe this audio content218, but when the third participant talks about other topics (e.g., the weather), the transcriptor200does transcribe this audio content218. The participant10cmay similarly set time boundaries to that the transcriptor200does not memorialize any audio content218for a length of time, e.g., the next 2-minutes.

FIGS.2A and2Bare examples of the transcriptor200. The transcriptor200generally includes a diarization module220and an ASR module230(e.g., an AVASR module). The diarization module220is configured to receive audio data218that corresponds to the utterances12from the participants10of the communication session (e.g., captured by the audio capturing device216a) and image data217representing faces of the participants10of the communication session, segment the audio data218into a plurality of segments222,222a-n(e.g., fixed-length segments or variable-length segments), and generate diarization results224that include a corresponding speaker label226assigned to each segment222using a probability model (e.g., a probabilistic generative model) based on the audio data218and the image data217. In other words, the diarization module220includes a series of speaker recognition tasks with short utterances (e.g., segments222) and determines whether two segments222of a given conversation were spoken by the same participant10. Simultaneously, the diarization module220may execute a face tracking routine to identify which participant10is speaking during which segment222to further optimize speaker recognition. The diarization module220is then configured to repeat the process for all segments222of the conversation. Here, the diarization results224provide time-stamped speaker labels226,226a-efor the received audio data218that not only identify who is speaking during a given segment222, but also identify when speaker changes occur between adjacent segments222. Here, the speaker labels226may serve as identifiers204within the transcript202.

In some examples, the transcriptor200receives the privacy request14at the diarization module220. Since the diarization module220identifies speaker labels226or identifiers204, the diarization module220may advantageously resolve a privacy request14that corresponds to an identity-based privacy request14. In other words, the diarization module220receives a privacy request14when the privacy request14requests not to identify a participant10by an identifier204such as the label226when the participant10is the speaker. When the diarization module220receives a privacy request14, the diarization module220is configured to determine whether the participant corresponding to the request14matches the label226generated for a given segment222. In some examples, an image of the participant's10face may be used to associate the participant10with the label226for that participant10. When the label226for a segment222matches an identity of the participant10corresponding to the request14, the diarization module220may prevent the transcriptor200from applying the label226or identifier204to a corresponding portion of the resulting transcription202that transcribes the particular segment222into text. When the label226for a segment222fails to match an identity of the participant10corresponding to the request14, the diarization module220may allow the transcriptor to apply the label226and identifier204to the portion of the resulting transcription202that transcribes the particular segment into text. In some implementations, when the diarization module220receives the request14, the ASR module230is configured to wait to transcribe the audio data218from utterances12. In other implementations, the ASR module230transcribes in real-time and the resulting transcription202removes labels226and identifiers204in real-time for any participants10providing privacy requests14opting out from having their speech information transcribed. Optionally, the diarization module220may further distort the audio data218associated with these participants10seeking privacy so that their speaking voice is altered in a way that cannot be used to identify the participant10.

The ASR module230is configured to receive the audio data218corresponding to the utterances12and the image data217representing faces of the participants10while speaking the utterances12. Using the image data217, the ASR module230transcribes the audio data218into corresponding ASR results232. Here, the ASR result232refers to a textual transcription of the audio data218(e.g., the transcript202). In some examples, the ASR module230communicates with the diarization module220to utilize the diarization results224associated with the audio data218for improving speech recognition based on utterances12. For instance, the ASR module230may apply different speech recognition models (e.g., language models, prosody models) for different speakers identified from the diarization results224. Additionally or alternatively, the ASR module230and/or the diarization module220(or some other component of the transcriptor200) may index a transcription232of the audio data218using the time-stamped speaker labels226predicted for each segment222obtained from the diarization results224. Stated differently, the ASR module230uses the speaker labels226from the diarization module220to generate the identifiers204for speakers within the transcript202. As shown inFIGS.1A-1E, the transcript202for a communication session within the environment100may be indexed by speaker/participant10to associate portions of the transcript202with the respective speaker/participant10in order to identify what each speaker/participant10said.

In some configurations, the ASR module230receives the privacy request14for the transcriptor200. For example, the ASR module230receives the privacy request14for the transcriptor200whenever the privacy request14corresponds to a request14not to transcribe speech for a particular participant10. In other words, the ASR module230may receive the privacy request14whenever the request14is not a label/identifier based privacy request14. In some examples, when the ASR module230receives a privacy request14, the ASR module230first identifies the participant10corresponding to the privacy request14based on the speaker labels226determined by the diarization module220. Then, when the ASR module230encounters speech to be transcribed for that participant10, the ARS module230applies the privacy request14. For instance, when the privacy request14requests not to transcribe speech for that particular participant10, the ASR module230does not transcribe any speech for that participant and waits for speech to occur by a different participant10.

Referring toFIG.2B, in some implementations, the transcriptor200includes a detector240for executing the face tracking routine. In these implementations, the transcriptor200first processes the audio data218to generate one or more candidate identities for the speaker. For example, for each segment222, the diarization module220may include multiple labels226,226a1-3as candidate identities for the speaker. In other words, the model may be a probability model that outputs multiple labels226,226a1-3for each segment222where each label226of the multiple labels226,226a1-3is a potential candidate that identifies the speaker. Here, the detector240of the transcriptor200uses the images217,217a-ncaptured by the image capturing device216bto determine which candidate identity had the best visual features indicating that he or she is the speaker of a particular segment22. In some configurations, the detector240generates a score242for each candidate identity where the score242indicates a confidence level that the candidate identity is the speaker based on the association between the audio signal (e.g., audio data218) and the visual signal (e.g., the captured images217a-n). Here, the highest score242may indicate the greatest likelihood that the candidate identity is the speaker. InFIG.2B, the diarization module220generates three labels226a1-3at a particular segment222. The detector240generates a score242for each of these labels226(e.g., shown as three scores2421-3) based on images217from the time in the audio data218where the segment222occurs. Here,FIG.2Bindicates the highest score242by a bolded square around the third label226a3associated with the third score2423. When the transcriptor200includes the detector240, the best candidate identity may be communicated to the ASR module230to form the identifier204of the transcript202.

Additionally or alternatively, the process may be reversed such that the transcriptor200first processes the image data217to generate one or more candidate identities for the speaker based on the image data217. Then for each candidate identity, the detector240generates a confidence score242indicating a likelihood that a face of the corresponding candidate identity includes a speaking face for a corresponding segment222of audio data218. For instance, the confidence score242for each candidate identity indicates a likelihood that a face of the corresponding candidate identity includes a speaking face during the image data217corresponding to an instance of time for a segment222of the audio data218. In other words, for each segment222, the detector240may score242whether image data217corresponding to participants10has facial expressions similar to or matching the facial expressions of a speaking face. Here, the detector240selects the identity of the speaker of the corresponding segment of the audio data218with the highest confidence score242as the candidate identity.

In some examples, the detector240is part of the ASR module230. Here, the ASR module230executes the face tracking routine by implementing an encoder frontend having an attention layer configured to receive a plurality of video tracks217a-nof the image data217, whereby each video track is associated with a face of a respective participant. In these examples, the attention layer at the ASR module230is configured to determine a confidence score indicating a likelihood that the face of the respective person associated with the video face track includes a speaking face of the audio track. Additional concepts and features related to an audio-visual ASR module including an encoder front end having an attention layer for multi-speaker ASR recognition can be found in U.S. Provisional Patent Application 62/923,096, filed on Oct. 18, 2019, which is hereby incorporated by reference in its entirety.

In some configurations, the transcriptor200(e.g., at the ASR module230) is configured to support a multi-lingual environment100. For example, when the transcriptor200generates the transcript202, the transcriptor200is capable of generating the transcript202in different languages. This feature may enable the environment100to include a remote location that has one or more participants10that speak a different language than the host location. Moreover, in some situations, the speaker in a meeting may be a non-native speaker or a speaker where the language of the meeting is not the first language of the speaker. Here, a transcript202of the content from the speaker may assist other participants10in the meeting to understand the presented content. Additionally or alternatively, the transcriptor200may be used to provide a speaker with feedback on his or her pronunciation. Here, by combining the video and/or the audio data, the transcriptor200may indicate an incorrect pronunciation (e.g., allowing a speaker to learn and/or to adapt with the help of the transcriptor200). As such, the transcriptor200may provide a notification to the speaker that provides the feedback on his/her pronunciation

FIG.3is an example arrangement of operations for a method300of transcribing content (e.g., at the data processing hardware212of the transcriptor200). At operation302, the method300includes receiving an audio-visual signal217,218including audio data218and image data217. The audio data218corresponds to speech utterances12from a plurality of participants10,10a-nin a speech environment100and the image data217represents faces of the plurality of participants10in the speech environment100. At operation304, the method300includes receiving a privacy request14from a participant10of the plurality of participants10a-n. The privacy request14indicates a privacy condition associated with the participant10in the speech environment100. At operation306, the method300segments the audio data218into a plurality of segments222,222a-n. At operation308, the method300includes performing operations308,308a-cfor each segment222of the audio data218. At operation308a, for each segment222of the audio data218, the method300includes determining from among the plurality of participants10a-n, an identity of a speaker of a corresponding segment222of the audio data218based on the image data217. At operation308b, for each segment222of the audio data218, the method300includes determining whether the identity of the speaker of the corresponding segment222includes the participant10associated with the privacy condition indicated by the received privacy request14. At operation308c, for each segment222of the audio data218, when the identity of the speaker of the corresponding segment222includes the participant10, the method300includes applying the privacy condition to the corresponding segment222. At operation310, the method300includes processing the plurality of segments222a-nof the audio data218to determine a transcript202for the audio data218.

In situations in which certain implementations discussed herein may collect or use personal information about users (e.g., user data extracted from other electronic communications, information about a user's social network, a user's location, a user's time, a user's biometric information, and a user's activities and demographic information, relationships between users, etc.), users are provided with one or more opportunities to control whether information is collected, whether the personal information is stored, whether the personal information is used, and how the information is collected about the user, stored and used. That is, the systems and methods discussed herein collect, store and/or use user personal information only upon receiving explicit authorization from the relevant users to do so.

For example, a user is provided with control over whether programs or features collect user information about that particular user or other users relevant to the program or feature. Each user for which personal information is to be collected is presented with one or more options to allow control over the information collection relevant to that user, to provide permission or authorization as to whether the information is collected and as to which portions of the information are to be collected. For example, users can be provided with one or more such control options over a communication network. In addition, certain data may be treated in one or more ways before it is stored or used so that personally identifiable information is removed. As one example, a user's identity may be treated so that no personally identifiable information can be determined.

FIG.4is schematic view of an example computing device400that may be used to implement the systems and methods described in this document. The computing device400is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The components shown here, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations described and/or claimed in this document.

The computing device400includes a processor410(e.g., data processing hardware), memory420(e.g., memory hardware), a storage device430, a high-speed interface/controller440connecting to the memory420and high-speed expansion ports450, and a low speed interface/controller460connecting to a low speed bus470and a storage device430. Each of the components410,420,430,440,450, and460, are interconnected using various busses, and may be mounted on a common motherboard or in other manners as appropriate. The processor410can process instructions for execution within the computing device400, including instructions stored in the memory420or on the storage device430to display graphical information for a graphical user interface (GUI) on an external input/output device, such as display480coupled to high speed interface440. In other implementations, multiple processors and/or multiple buses may be used, as appropriate, along with multiple memories and types of memory. Also, multiple computing devices400may be connected, with each device providing portions of the necessary operations (e.g., as a server bank, a group of blade servers, or a multi-processor system).

The memory420stores information non-transitorily within the computing device400. The memory420may be a computer-readable medium, a volatile memory unit(s), or non-volatile memory unit(s). The non-transitory memory420may be physical devices used to store programs (e.g., sequences of instructions) or data (e.g., program state information) on a temporary or permanent basis for use by the computing device400. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM)/programmable read-only memory (PROM)/erasable programmable read-only memory (EPROM)/electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware, such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM) as well as disks or tapes.

The storage device430is capable of providing mass storage for the computing device400. In some implementations, the storage device430is a computer-readable medium. In various different implementations, the storage device430may be a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid state memory device, or an array of devices, including devices in a storage area network or other configurations. In additional implementations, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer- or machine-readable medium, such as the memory420, the storage device430, or memory on processor410.

The high speed controller440manages bandwidth-intensive operations for the computing device400, while the low speed controller460manages lower bandwidth-intensive operations. Such allocation of duties is exemplary only. In some implementations, the high-speed controller440is coupled to the memory420, the display480(e.g., through a graphics processor or accelerator), and to the high-speed expansion ports450, which may accept various expansion cards (not shown). In some implementations, the low-speed controller460is coupled to the storage device430and a low-speed expansion port470. The low-speed expansion port470, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), may be coupled to one or more input/output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, e.g., through a network adapter.

The computing device400may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a standard server400aor multiple times in a group of such servers400a, as a laptop computer400b, or as part of a rack server system400c.

Various implementations of the systems and techniques described herein can be realized in digital electronic and/or optical circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

These computer programs (also known as programs, software, software applications or code) include machine instructions for a programmable processor, and can be implemented in a high-level procedural and/or object-oriented programming language, and/or in assembly/machine language. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, non-transitory computer readable medium, apparatus and/or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term “machine-readable signal” refers to any signal used to provide machine instructions and/or data to a programmable processor.

The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a processor for performing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

To provide for interaction with a user, one or more aspects of the disclosure can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen for displaying information to the user and optionally a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user's client device in response to requests received from the web browser.

A number of implementations have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the disclosure. Accordingly, other implementations are within the scope of the following claims.