Patent Description:
Speech transcription technology can provide machine aided transcription of spoken word to text. The use of speech transcription technology in an aviation environment, however, can be challenging. When air traffic is high, there can be multiple speakers on a communication channel including multiple controllers and multiple traffic pilots. Channel noise along with multiple speakers including different ATCs, traffic pilots and own ship pilots in a communication radio channel can pose a huge challenge for an ATC transcription system to produce reliable and accurate results. Different accents spoken by the multiple speakers can make it more difficult for an ATC transcription system to produce reliable and accurate results. Missing call signs in an ATC conversation can add to confusion.

Hence, it is desirable to provide speech transcription systems and methods for reliably and accurately transcribing over-the-air conversations between ATC and pilots while accounting for difficulties presented in an aviation environment. Furthermore, other desirable features and characteristics of the present invention will become apparent from the subsequent detailed description and the appended claims, taken in conjunction with the accompanying drawings and the foregoing technical field and background.

Documents cited during prosecution include <CIT>; <CIT>; and <CIT>.

Aspects and preferred embodiments of the invention are defined in the appended claims. Disclosed herein is a flight deck system for extracting speaker information in an ATC (Air Traffic Controller) conversation and displaying the speaker information on a graphical display unit is disclosed. The flight deck system includes a controller configured to: segment a stream of audio received over radio from an ATC and other aircraft into a plurality of chunks, wherein each chunk has a speaker; for each chunk, determine if the speaker for the chunk is enrolled as a speaker in a speaker database; when the speaker for the chunk is enrolled as a speaker in the speaker database, decode the chunk using a speaker-dependent automatic speech recognition (ASR) model that is specific for the speaker and tag the chunk with a permanent name for the speaker; when the speaker for the chunk is not enrolled as a speaker in the speaker database, assign a temporary name for the speaker of the chunk, tag the chunk with the temporary name, and decode the chunk using a speaker independent speech recognition model; format the decoded chunk as text; and signal the graphical display unit to display the formatted text along with an identity for the speaker of the formatted text, wherein the identity includes the permanent name of the speaker or the temporary name assigned to the speaker. The system is further configured to enroll a non-enrolled speaker into the speaker database and create a speaker-dependent ASR model for the non-enrolled speaker after a predetermined number of chunks of audio from the non-enrolled speaker are received.

Also disclosed herein is a method in a flight deck system for extracting speaker information in an ATC (Air Traffic Controller) conversation and displaying the speaker information on a graphical display unit is disclosed. The method includes: segmenting a stream of audio received over radio from an ATC and other aircraft into a plurality of chunks, wherein each chunk has a speaker; for each chunk, determining if the speaker for the chunk is enrolled as a speaker in a speaker database; when the speaker for the chunk is enrolled as a speaker in the speaker database, decoding the chunk using a speaker-dependent automatic speech recognition (ASR) model that is specific for the speaker and tagging the chunk with a permanent name for the speaker; when the speaker for the chunk is not enrolled as a speaker in the speaker database, assigning a temporary name for the speaker of the chunk, tagging the chunk with the temporary name, and decoding the chunk using a speaker independent speech recognition model; formatting the decoded chunk as text; and signaling the graphical display unit to display the formatted text along with an identity for the speaker of the formatted text, the identity including the permanent name of the speaker or the temporary name assigned to the speaker. The method further includes enrolling a non-enrolled speaker into the speaker database and creating a speaker-dependent ASR model for the non-enrolled speaker after a predetermined number of chunks of audio from the non-enrolled speaker are received.

Also disclosed herein is a non-transitory computer-readable medium having stored thereon instructions which when executed by one or more processors in a flight deck system causes the flight deck system to perform a method for extracting speaker information in an ATC (Air Traffic Controller) conversation and displaying the speaker information on a graphical display unit is disclosed. The method includes: segmenting a stream of audio received over radio from an ATC and other aircraft into a plurality of chunks, wherein each chunk has a speaker; for each chunk, determining if the speaker for the chunk is enrolled as a speaker in a speaker database; when the speaker for the chunk is enrolled as a speaker in the speaker database, decoding the chunk using a speaker-dependent automatic speech recognition (ASR) model that is specific for the speaker and tagging the chunk with a permanent name for the speaker; when the speaker for the chunk is not enrolled as a speaker in the speaker database, assigning a temporary name for the speaker of the chunk, tagging the chunk with the temporary name, and decoding the chunk using a speaker independent speech recognition model; formatting the decoded chunk as text; and signaling the graphical display unit to display the formatted text along with an identity for the speaker of the formatted text, the identity including the permanent name of the speaker or the temporary name assigned to the speaker. The method further includes enrolling a non-enrolled speaker into the speaker database and creating a speaker-dependent ASR model for the non-enrolled speaker after a predetermined number of chunks of audio from the non-enrolled speaker are received.

Embodiments of the present disclosure may be described herein in terms of functional and/or logical components and various processing steps. It should be appreciated that such functional and/or logical components may be realized by any number of hardware, software, and/or firmware components configured to perform the specified functions.

Speech transcription technology can provide machine aided transcription of spoken word to text. This can be very helpful in an aviation environment wherein machine aided transcription systems and methods can help a flight crew member (e.g., pilot) more quickly and better understand over-the-air communication between flight crew members (both own ship flight crew members and traffic flight crew members) and one or more air traffic controllers, especially when one or more speakers speaks with an accent and/or when communications take place in a noisy environment.

The subject matter described herein discloses apparatus, systems, techniques, and articles for identifying and classifying messages meant for own ship and for identifying and classifying messages from or directed to traffic aircraft. The apparatus, systems, techniques, and articles provided herein can contribute to more effective communication by reducing the frequency with which a pilot may have to ask a controller to clarify a misunderstood message. The apparatus, systems, techniques, and articles provided herein can assist a pilot with understanding a message from a non-native English speaker thereby reducing a pilot's workload (e.g., the workload while flying near airports can be high and during that time trying to comprehend accented ATC commands along with the radio noise can increase a pilot's workload considerably). The apparatus, systems, techniques, and articles provided herein can provide a Quick Reference Tool that can help Pilots adhere accurately to strategic ATC commands without having to remember each command. The apparatus, systems, techniques, and articles provided herein can identify the speaker in an ATC pilot conversation therefore allowing the use of a speaker-dependent speech recognition system to improve transcription accuracy. The apparatus, systems, techniques, and articles provided herein can utilize natural language processing (NLP) techniques to further refine transcription output.

<FIG> is a block diagram depicting an example flight environment <NUM> such as one around a busy aerodrome. The example environment <NUM> includes a plurality of aerial vehicles (own ship aircraft <NUM> and traffic aircraft <NUM>, <NUM> in this example), but could include a variety of types of aerial vehicles such as helicopters, UAVs (unmanned aerial vehicles), and others. The example environment <NUM> also includes a plurality of air traffic control towers <NUM>, <NUM> containing air traffic controllers (ATC) for directing ground and air traffic in the vicinity of the aerial vehicles.

The example own ship aircraft <NUM> includes avionics equipment <NUM> that receives the ongoing communications between the aerial vehicles (e.g., <NUM>, <NUM>, <NUM>) and ATC (e.g., via towers <NUM>, <NUM>) using communication equipment <NUM>, and presents the ongoing communications as a continuous stream of audio <NUM> to an ATC transcription system <NUM>. The ATC transcription system <NUM> decodes the continuous stream of audio <NUM>, generates formatted text from the decoded continuous stream of audio, and signals a cockpit display device <NUM> to display the generated formatted text along with an identity for the speaker of the formatted text for viewing by the flight crew onboard the own ship aircraft <NUM>. The cockpit display device <NUM> may be one of many types of graphical display units onboard an aircraft such as a navigation display, a PFD (primary flight display), a PED (personal electronic device), an EFB (electronic flight bag), HUD (heads up display), HDD (heads down display), and others. The display of the formatted text may be made via a graphical display page <NUM> that displays each generated textual message (M1, M2, M3, M4,. ) in a manner that visually identifies which messages are directed to the own ship aircraft.

<FIG> is a block diagram depicting example communication equipment <NUM> and an example ATC transcription system <NUM> for receiving ongoing over-the-air communications and generating formatted text from the over-the-air communications for display onboard an aerial vehicle. The example communication equipment <NUM> includes a Com radio <NUM> (such as that known in the art) and an intercom channel <NUM> (such as that known in the art) for receiving over-the-air communications between various aerial vehicles (own ship and traffic aerial vehicles) and ATC. The example communication equipment <NUM> also includes an audio panel <NUM> (such as that known in the art) for accumulating the over-the-air communications from various sources and outputting the audio from the over-the-air communications.

The example ATC transcription system <NUM> is configured to receive a continuous stream of audio from the communication equipment <NUM> (e.g., via audio panel <NUM>) and generate formatted text therefrom for display onboard the aerial vehicle. The example ATC transcription system <NUM> includes a voice activity detection module <NUM>, a speaker identification module <NUM>, a speech recognizer module <NUM>, and a text formatter module <NUM>, such as an NLP (natural language processor) based text formatter module, an expert based text formatter module, or a rule based text formatter module.

Each of the voice activity detection module <NUM>, speaker identification module <NUM>, speech recognizer module <NUM>, and text formatter module <NUM> is implemented by a processing component such as a controller (e.g., the same or separate controllers). The processing component includes at least one processor and a computer-readable storage device or media encoded with programming instructions for configuring the processing component. The processor may be any custom-made or commercially available processor, a central processing unit (CPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), an auxiliary processor among several processors associated with the processing component, a semiconductor-based microprocessor (in the form of a microchip or chip set), any combination thereof, or generally any device for executing instructions.

The computer readable storage device or media may include volatile and nonvolatile storage in read-only memory (ROM), random-access memory (RAM), and keep-alive memory (KAM), for example. KAM is a persistent or non-volatile memory that may be used to store various operating variables while the processor is powered down. The computer-readable storage device or media may be implemented using any of a number of known memory devices such as PROMs (programmable read-only memory), EPROMs (electrically PROM), EEPROMs (electrically erasable PROM), flash memory, or any other electric, magnetic, optical, or combination memory devices capable of storing data, some of which represent executable programming instructions, used by the processing component.

The example voice activity detection module <NUM> is configured to segment the received stream of audio into a plurality of chunks, wherein each chunk has a speaker. To segment the received audio into a plurality of chunks the example voice activity detection module <NUM> may access a voice activity detection database <NUM> to obtain rules for marking voiced and non-voiced boundaries in the received stream of audio and segmenting the received stream of audio into a plurality of chunks based on sentence or phrase boundaries derived from the voiced and non-voiced boundaries.

The example speaker identification module <NUM> is configured to determine the speaker for each chunk. The example speaker identification module <NUM> determines if the speaker for a chunk is enrolled as a speaker in an enrolled speaker database <NUM>. When the speaker for the chunk is enrolled as a speaker in the enrolled speaker database <NUM>, the example speaker identification module <NUM> tags the chunk with a permanent name for the speaker. When the speaker for the chunk is not enrolled as a speaker in the enrolled speaker database <NUM>, the example speaker identification module <NUM> tags the chunk with a temporary name for the speaker. The example speaker identification module <NUM> optionally determines the language spoken by the speaker for the chunk and tags the chunk with the language.

To determine if the speaker for the chunk is enrolled as a speaker, the example speaker identification module <NUM> is configured extract speaker-based features from the audio (such as speaker embeddings, or vocal cord and prosody based features like pitch, articulation, speed, modulation, etc.); generate a plurality of similarity scores for the extracted speaker-based features based on a comparison of the extracted speaker-based features with a plurality of model files in the enrolled speaker database <NUM> for a plurality of speakers from the enrolled speaker database <NUM>; and when a specific similarity score determined based on the comparison of the extracted speaker-based features to a model file for a particular speaker in the enrolled speaker database <NUM> exceeds a threshold level, associate the chunk with the particular speaker.

The example speaker identification module <NUM> is further configured to save the extracted speaker-based features for the chunk as a model file for a speaker in the enrolled speaker database <NUM> with a temporary name when none of the plurality of similarity scores for the extracted speaker-based features exceeds the threshold level. The model file for the extracted speaker-based features for a speaker with a temporary name can be saved until a predetermined number of chunks of audio are reached.

The example speaker identification module <NUM> is further configured to enroll previously unenrolled speakers into the enrolled speaker database <NUM>. When the speaker for a chunk is not enrolled as a speaker in the enrolled speaker database <NUM>, the speaker identification module <NUM> is configured to: accumulate a predetermined number of chunks of audio that are tagged with the same temporary name, and when the predetermined number of chunks is reached, generate a speaker dependent ASR model for the speaker with the temporary name using the predetermined number of accumulated chunks, store the speaker dependent ASR model in an acoustic database <NUM>, and enroll the speaker with the temporary name along with the corresponding extracted speaker-based features as a model file in the enrolled speaker database <NUM>.

To accumulate a predetermined number of chunks of audio that are tagged with the same temporary name, the example speaker identification module <NUM> is configured to extract speaker-based features by taking a portion of a chunk of audio with a duration length set by pre-determined threshold (e.g., <NUM> sec but this threshold can be optimized to a lower value) and processing the portion to create a vector that consist of speaker characteristics, such as speaker embeddings, or vocal cord and prosody based features like pitch, articulation, speed, modulation, etc. This speaker vector is unique and can contain speaker features regarding accent/dialect, speech rate, pitch, tempo, articulation activity, pauses, phonetic variations, etc. associated with the speaker. The name of each speaker along with its corresponding features are saved as a separate model file or alternatively combined based on similarity into one file. These model files are used by the speaker identification module <NUM> to identify the speaker based on input audio.

To accumulate a predetermined number of chunks of audio that are tagged with the same temporary name, the example speaker identification module <NUM> is further configured to generate a similarity score for the extracted speaker-based features based on a comparison of the extracted speaker-based features with the model file for the speaker with the temporary name; and when the similarity score determined based on the comparison of the extracted speaker-based features to the model file for the speaker with the temporary name exceeds a threshold level, associate the chunk with the speaker with the temporary name.

The example speaker identification module <NUM> may further allow a flight crew member (e.g., pilot) to generate a replacement name for a temporary name by editing an auto generated temporary name, assigning tags based on, for example, call sign, or simply assigning labels such as ATC1 or TP1 or traffic pilot1.

<FIG> is a diagram illustrating an example tagging of chunks of audio in a stream of audio <NUM>. In this example, four chunks of audio (<NUM>, <NUM>, <NUM>, <NUM>) are presented in the stream of audio <NUM>. The example voice activity detection module <NUM> is configured to segment the received stream of audio <NUM> into the four chunks (<NUM>, <NUM>, <NUM>, <NUM>). The example voice activity detection module <NUM> can identify boundaries (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>) for the chunks (<NUM>, <NUM>, <NUM>, <NUM>). The example speaker identification module <NUM> can then determine the speaker of each chunk (<NUM>, <NUM>, <NUM>, <NUM>). In this example, the first chunk <NUM> is bounded by boundaries (<NUM>, <NUM>) and is associated with Traffic Pilot <NUM> as the speaker; the second chunk <NUM> is bounded by boundaries (<NUM>, <NUM>) and is associated with ATC1 as the speaker; the third chunk <NUM> is bounded by boundaries (<NUM>, <NUM>) and is associated with Jim (ATC) as the speaker; and the fourth chunk <NUM> is bounded by boundaries (<NUM>, <NUM>) and is associated with James (own ship Pilot) as the speaker.

Referring again to <FIG>, the example speech recognizer module <NUM> is configured to decode the tagged chunks of audio using speech recognition techniques to convert the tagged chunks of audio into raw text. When the speaker for a chunk is enrolled as a speaker in the enrolled speaker database <NUM>, a corresponding speaker-dependent automatic speech recognition (ASR) model has been provided in the acoustic database <NUM>, and the example speech recognizer module <NUM> is configured to decode the chunk as raw text using a speaker-dependent ASR model that is specific for the speaker and optionally a language model (e.g., stored in speech database <NUM>) that relates to a group of words or vocabulary recognized from the chunk with its pronunciation or lexical variations. Speaker based models can provide better performance in terms of accuracy and speed in text decoding as compared to generic or speaker independent models. When the speaker for a chunk is not enrolled as a speaker in the enrolled speaker database <NUM>, a corresponding speaker-dependent ASR model has not been provided in the acoustic database <NUM>, and the example speech recognizer module <NUM> is configured to decode the chunk using a speaker-independent ASR model provided in the acoustic database <NUM> that is generic and optionally a language model (e.g., stored in speech database <NUM>). The speaker-dependent ASR model can be configured using known techniques in the art such as speaker adaptation. The speaker-based ASR models can be constrained with vocabulary that can be spoken by either ATC or flight crew members, so not only accuracy, but latency of speech recognition will improve.

The example speech recognizer module <NUM> may also be configured to decode the tagged chunks of audio using a language model in a speech database <NUM>. The language model may be used for the chunk to account for lexical variations in phrases that may be represented in the chunk. The example speech recognizer module <NUM> is configured to output raw text from its decoding operations.

The example speech recognizer module <NUM> may be implemented using machine learning techniques such as deep neural networks that can use speaker dependent and speaker independent models, statistical techniques that can use speaker dependent and speaker independent models, rule-based systems that can use speaker dependent and speaker independent models, and others.

An example NLP based text formatter module <NUM> is configured to format the raw text as formatted text using natural language processing techniques, such as deep learning techniques, text embeddings, machine translation, neural machine translation (NMT), recurrent neural network (RNN), long short-term memory (LTSM) networks, gated recurrent units (GRU), bidirectional encoder representations from transformers (BERT), generative pre-trained transformer (GPT), XLNET, and others.

The example text formatter module <NUM> is further configured to signal the graphical display unit to display the formatted text along with an identity for the speaker of the formatted text wherein the identity includes the permanent name of the speaker or the temporary name assigned to the speaker. The example text formatter module <NUM> may be further configured to signal the graphical display unit to display the identity for the speaker of the formatted text using the replacement name, when flight crew member initiated substitution of a temporary name with a flight crew member inputted replacement name has been received.

The example ATC transcription system <NUM> may continuously update the enrolled speaker database <NUM> with the enrollment of unknown speakers and the acoustic database <NUM> with speaker dependent speech recognition models, iteratively. The example ATC transcription system <NUM> can use the speaker identification information to filter out the transcription of conversations involving traffic pilots and/or specific ATC when needed to allow for the display and/or recording of conversations involving only the own ship pilot or optionally only with the own ship pilot and select traffic pilots.

The example ATC transcription system <NUM> may allow the own ship pilot to filter out messages for a selected traffic pilot or selected ATC based on speaker information. The example ATC transcription system <NUM> is configured to automatically enroll own ship pilots thus allowing the system to distinguish own ship pilots from other traffic pilots. The example ATC transcription system <NUM> is also configured to automatically store ATC speaker information along with flight path information and automatically enroll the ATCs the own ship is likely to encounter along the flight path. Because there is a lot of conversation between an own ship pilot and ATC, there should be enough voice data available to enroll different ATCs along various flight legs. During flight, a lot of conversations between a traffic pilot and an ATC may take place and the example ATC transcription system <NUM> is configured to mark voice messages by the traffic pilot as being from an unknown speaker with an arbitrary label such as X or Y. The example ATC transcription system <NUM> is configured to provide a user interface to allow the own ship pilot to edit arbitrary labels and relabel them with a label such as Traffic pilot1, TP1 or by callsign for traffic pilot messages and ATC1 for an ATC. This editing feature for labels will help pilots in future flights as these edited labels or tags are stored by the example ATC transcription system <NUM> with flight route information and GPS information. Consequently, the example ATC transcription system <NUM> can provide, for future flights on same route, various ATCs and scheduled traffic pilots as enrolled speakers. Also, this labeling can help in post flight analytics.

The example ATC transcription system <NUM> is configured to signal the graphical display unit to display, along with the formatted text and the identity (e.g., automatically generated and pilot edited speaker tags) for the speaker of the formatted text, extracted information including count and duration of messages for the speaker of the formatted text, number of total speakers during flight journey, percentage of messages in the flight journey for the speaker of the formatted text, and other extracted information.

The output of the example ATC transcription system <NUM> can be displayed based on speaker tagged information in different colors, shapes, or font sizes (visual attributes). The speaker information logged by the example ATC transcription system <NUM> may be useful for post flight analysis and recreating a flight scenario.

<FIG> is a process flow chart depicting an example process <NUM> for extracting speaker information in an ATC transcription and generating speaker information for display to flight crew. The order of operation within the process <NUM> is not limited to the sequential execution as illustrated in the figure but may be performed in one or more varying orders as applicable and in accordance with the present disclosure.

The example process <NUM> includes voice activity detection (VAD) wherein a VAD module marks the voice and non-voiced boundaries from received audio (operation <NUM>), segmenting the audio in chunks based on sentence/phrase boundary derived from voiced and non-voiced boundaries of VAD (operation <NUM>) and processing each chunk separately (operation <NUM>).

The example process <NUM> includes identifying, via a speaker identification module, enrolled speaker and newly enrolled unidentified speakers (in the speaker database <NUM>) from the given audio (operation <NUM>).

The example process <NUM> includes determining whether an enrolled speaker is found (decision <NUM>). If an enrolled speaker is found, yes at decision <NUM>, then the example process <NUM> includes decoding audio into raw text using a speaker-based ASR model <NUM> (operation <NUM>) and formatting the raw text into formatted text using natural language processing (operation <NUM>).

If an enrolled speaker is not found, no add decision for <NUM>, then the example process <NUM> includes tagging audio as having an unknown speaker (such as speaker X, speaker Y, etc.) (operation <NUM>), accumulating the audio of an unknown speaker and using it for speaker enrollment and to create specialized speech models <NUM> for the speaker (operation <NUM>), decoding the audio into raw text using a generic speech model <NUM> (operation <NUM>), and formatting the raw text into formatted text using natural language processing (operation <NUM>).

The example process <NUM> also includes enrolling the audio of an unenrolled speaker to create and/or update a speaker database <NUM> (operation <NUM>). External speaker data <NUM> may be used to enroll audio of a speaker to create the speaker database. The example process <NUM> ends after formatting text using natural language processing.

<FIG> is a process flow chart depicting an example process <NUM> for extracting speaker information in an ATC transcription and displaying the speaker information on a graphical display unit. The order of operation within the process <NUM> is not limited to the sequential execution as illustrated in the figure but may be performed in one or more varying orders as applicable and in accordance with the present disclosure.

The example process <NUM> includes receiving a stream of audio over radio from ATC and other aircraft (operation <NUM>) and segmenting the received stream of audio into a plurality of chunks, wherein each chunk has a speaker (operation <NUM>). The segmenting the received audio into a plurality of chunks may comprise marking voiced and non-voiced boundaries in the received stream of audio and segmenting the received stream of audio into a plurality of chunks based on sentence or phrase boundaries derived from the voiced and non-voiced boundaries.

The example process <NUM> includes determining, for each chunk, if the speaker for the chunk is enrolled as a speaker in a speaker database (decision <NUM>). Determining if the speaker for the chunk is enrolled as a speaker may comprise: extracting speaker-based features from the audio; generating a plurality of similarity scores for the extracted speaker-based features based on a comparison of the extracted speaker-based features with a plurality of model files for a plurality of speakers; and when a specific similarity score determined based on the comparison of the extracted speaker-based features to a model file for a particular speaker exceeds a threshold level, associating the chunk with the particular speaker.

When the speaker for the chunk is enrolled as a speaker in a speaker database (yes at decision <NUM>), the example process <NUM> includes decoding the chunk using a speaker-dependent automatic speech recognition (ASR) model that is specific for the speaker and tag the chunk with a permanent name for the speaker (operation <NUM>). The decoding may comprise decoding the chunk as raw text.

When the speaker for the chunk is not enrolled as a speaker in a speaker database (no at decision <NUM>), the example process <NUM> includes assigning a temporary name for the speaker of the chunk, tag the chunk with the temporary name, and decoding the chunk using a speaker independent speech recognition model (operation <NUM>). When the speaker for a chunk is not enrolled as a speaker in the speaker database, the method may further comprise: accumulating a predetermined number of chunks of audio that are tagged with the same temporary name; and when the predetermined number of chunks is reached, generating a speaker dependent ASR model for the speaker with the temporary name using the predetermined number of accumulated chunks; and enrolling the speaker with the temporary name and the speaker dependent ASR model for the speaker with the temporary name in the speaker database. The method may further allow a flight crew member (e.g., pilot) to generate a replacement name by editing an auto generated temporary name, assigning tags based on, for example, call sign, or simply assigning labels such as ATC1 or TP1 or traffic pilot1.

The example process <NUM> includes formatting the decoded chunk as text (operation <NUM>). The formatting the decoded chunk may comprise formatting raw text as formatted text using natural language processing.

The example process <NUM> includes signaling a graphical display unit to display the formatted text along with an identity for the speaker of the formatted text, the identity comprising the permanent name of the speaker or the temporary name assigned to the speaker (operation <NUM>). The method may further include receiving flight crew member initiated substitution of a temporary name with a flight crew member inputted replacement name and signaling the graphical display unit to display the identity for the speaker of the formatted text using the replacement name.

In one embodiment, a flight deck system for extracting speaker information in an ATC (Air Traffic Controller) conversation and displaying the speaker information on a graphical display unit is provided. The flight deck system comprises a controller configured to: segment a stream of audio received over radio from an ATC and other aircraft into a plurality of chunks, wherein each chunk has a speaker; for each chunk, determine if the speaker for the chunk is enrolled as a speaker in a speaker database; when the speaker for the chunk is enrolled as a speaker in the speaker database, decode the chunk using a speaker-dependent automatic speech recognition (ASR) model that is specific for the speaker and tag the chunk with a permanent name for the speaker; when the speaker for the chunk is not enrolled as a speaker in the speaker database, assign a temporary name for the speaker of the chunk, tag the chunk with the temporary name, and decode the chunk using a speaker independent speech recognition model; format the decoded chunk as text; and signal the graphical display unit to display the formatted text along with an identity for the speaker of the formatted text, wherein the identity comprises the permanent name of the speaker or the temporary name assigned to the speaker. The system is further configured to enroll a non-enrolled speaker into the speaker database and create a speaker-dependent ASR model for the non-enrolled speaker after a predetermined number of chunks of audio from the non-enrolled speaker are received.

These aspects and other embodiments may include one or more of the following features. The controller may be further configured to receive flight crew member initiated substitution of a temporary name with a flight crew member inputted replacement name; and signal the graphical display unit to display the identity for the speaker of the formatted text using the replacement name. To segment the received audio into a plurality of chunks the controller may be configured to: mark voiced and non-voiced boundaries in the received stream of audio; and segment the received stream of audio into a plurality of chunks based on sentence or phrase boundaries derived from the voiced and non-voiced boundaries. When the speaker for a chunk is not enrolled as a speaker in the speaker database, the controller may be further configured to: accumulate a predetermined number of chunks of audio that are tagged with the same temporary name; and when the predetermined number of chunks is reached, generate a speaker dependent ASR model for the speaker with the temporary name using the predetermined number of accumulated chunks; and enroll the speaker with the temporary name and the speaker dependent ASR model for the speaker with the temporary name in the speaker database. To decode the chunk, the controller may be configured to decode the chunk as raw text. To format the decoded chunk the controller may be configured to format the raw text as formatted text using natural language processing. The controller may be configured to generate speaker dependent ASR by speaker adaptation. To determine if the speaker for the chunk is enrolled as a speaker the controller may be configured to: extract speaker-based features from the audio; generate a plurality of similarity scores for the extracted speaker-based features based on a comparison of the extracted speaker-based features with a plurality of model files for a plurality of speakers; and when a specific similarity score determined based on the comparison of the extracted speaker-based features to a model file for a particular speaker exceeds a threshold level, associate the chunk with the particular speaker. The controller may be further configured to save the extracted speaker-based features for the chunk as a model file for a speaker with a temporary name when none of the plurality of similarity scores for the extracted speaker-based features exceeds the threshold level. To accumulate a predetermined number of chunks of audio that are tagged with the same temporary name the controller may be configured to: generate a similarity score for the extracted speaker-based features based on a comparison of the extracted speaker-based features with the model file for the speaker with the temporary name; and when the similarity score determined based on the comparison of the extracted speaker-based features to the model file for the speaker with the temporary name exceeds a threshold level, associate the chunk with the speaker with the temporary name. The controller may be further configured to update the speaker database with the enrollment of unknown speakers and speaker dependent speech recognition models iteratively. To decode the chunk the controller may be configured to decode the chunk as raw text and to format the decoded chunk the controller may be configured to format the raw text as formatted text using natural language processing, an expert system, or a rule-based system. The controller may be further configured to signal the graphical display unit to display, along with the formatted text and the identity for the speaker of the formatted text, extracted information including count and duration of messages for the speaker of the formatted text, number of total speakers during flight journey, and percentage of messages in the flight journey for the speaker of the formatted text. To signal the graphical display unit to display the formatted text along with an identity for the speaker of the formatted text, the controller may be configured to: filter out ATC conversations with traffic aircraft based on flight crew member inputted filter criteria; and signal the graphical display unit to display the formatted text along with an identity for the speaker of the formatted text for ATC conversations that were not filtered out.

In another embodiment, a method in a flight deck system for extracting speaker information in an ATC (Air Traffic Controller) conversation and displaying the speaker information on a graphical display unit is provided. The method comprises: segmenting a stream of audio received over radio from an ATC and other aircraft into a plurality of chunks, wherein each chunk has a speaker; for each chunk, determining if the speaker for the chunk is enrolled as a speaker in a speaker database; when the speaker for the chunk is enrolled as a speaker in the speaker database, decoding the chunk using a speaker-dependent automatic speech recognition (ASR) model that is specific for the speaker and tagging the chunk with a permanent name for the speaker; when the speaker for the chunk is not enrolled as a speaker in the speaker database, assigning a temporary name for the speaker of the chunk, tagging the chunk with the temporary name, and decoding the chunk using a speaker independent speech recognition model; formatting the decoded chunk as text; and signaling the graphical display unit to display the formatted text along with an identity for the speaker of the formatted text, the identity comprising the permanent name of the speaker or the temporary name assigned to the speaker. The method further comprises enrolling a non-enrolled speaker into the speaker database and creating a speaker-dependent ASR model for the non-enrolled speaker after a predetermined number of chunks of audio from the non-enrolled speaker are received.

These aspects and other embodiments may include one or more of the following features. The segmenting the received audio into a plurality of chunks may comprise marking voiced and non-voiced boundaries in the received stream of audio; and segmenting the received stream of audio into a plurality of chunks based on sentence or phrase boundaries derived from the voiced and non-voiced boundaries. When the speaker for a chunk is not enrolled as a speaker in the speaker database, the method may further comprise: accumulating a predetermined number of chunks of audio that are tagged with the same temporary name; and when the predetermined number of chunks is reached, generating a speaker dependent ASR model for the speaker with the temporary name using the predetermined number of accumulated chunks; and enrolling the speaker with the temporary name and the speaker dependent ASR model for the speaker with the temporary name in the speaker database. The decoding may comprise decoding the chunk as raw text. The formatting the decoded chunk may comprise formatting the raw text as formatted text using natural language processing. The determining if the speaker for the chunk is enrolled as a speaker may comprise: extracting speaker-based features from the audio; generating a plurality of similarity scores for the extracted speaker-based features based on a comparison of the extracted speaker-based features with a plurality of model files for a plurality of speakers; and when a specific similarity score determined based on the comparison of the extracted speaker-based features to a model file for a particular speaker exceeds a threshold level, associating the chunk with the particular speaker. The method may further comprise saving the extracted speaker-based features for the chunk as a model file for a speaker with a temporary name when none of the plurality of similarity scores for the extracted speaker-based features exceeds the threshold level. The accumulating a predetermined number of chunks of audio that are tagged with the same temporary name may comprise: generating a similarity score for the extracted speaker-based features based on a comparison of the extracted speaker-based features with the model file for the speaker with the temporary name; and when the similarity score determined based on the comparison of the extracted speaker-based features to the model file for the speaker with the temporary name exceeds a threshold level, associating the chunk with the speaker with the temporary name. The method may further comprise updating the speaker database with the enrollment of unknown speakers and speaker dependent speech recognition models iteratively. The method may further comprise receiving flight crew member initiated substitution of a temporary name with a flight crew member inputted replacement name; and signaling the graphical display unit to display the identity for the speaker of the formatted text using the replacement name. The decoding may comprise decoding the chunk as raw text; and the formatting the decoded chunk may comprise formatting the raw text as formatted text using natural language processing, an expert system, or a rule-based system. The method may further comprise signaling the graphical display unit to display, along with the formatted text and the identity for the speaker of the formatted text, extracted information including count and duration of messages for the speaker of the formatted text, number of total speakers during flight journey, and percentage of messages in the flight journey for the speaker of the formatted text. The signaling the graphical display unit to display the formatted text along with an identity for the speaker of the formatted text may further comprise: filtering out ATC conversations with traffic aircraft based on flight crew member inputted filter criteria; and signaling the graphical display unit to display the formatted text along with an identity for the speaker of the formatted text for ATC conversations that were not filtered out.

In another embodiment, provide is a non-transitory computer-readable medium having stored thereon instructions which when executed by one or more processors in a flight deck system causes the flight deck system to perform a method for extracting speaker information in an ATC (Air Traffic Controller) conversation and displaying the speaker information on a graphical display unit. The method comprises: segmenting a stream of audio received over radio from an ATC and other aircraft into a plurality of chunks, wherein each chunk has a speaker; for each chunk, determining if the speaker for the chunk is enrolled as a speaker in a speaker database; when the speaker for the chunk is enrolled as a speaker in the speaker database, decoding the chunk using a speaker-dependent automatic speech recognition (ASR) model that is specific for the speaker and tagging the chunk with a permanent name for the speaker; when the speaker for the chunk is not enrolled as a speaker in the speaker database, assigning a temporary name for the speaker of the chunk, tagging the chunk with the temporary name, and decoding the chunk using a speaker independent speech recognition model; formatting the decoded chunk as text; and signaling the graphical display unit to display the formatted text along with an identity for the speaker of the formatted text, the identity comprising the permanent name of the speaker or the temporary name assigned to the speaker. The method further comprises enrolling a non-enrolled speaker into the speaker database and creating a speaker-dependent ASR model for the non-enrolled speaker after a predetermined number of chunks of audio from the non-enrolled speaker are received.

In another embodiment, a flight deck system for extracting speaker information in an ATC (Air Traffic Controller) conversation and displaying the speaker information on a graphical display unit is provided. The flight deck system comprises a controller configured to: receive a continuous stream of audio over radio from ATC and other aircraft; mark voiced and non-voiced boundaries in the received audio; segment the received audio into a plurality of chunks based on sentence/phrase boundaries derived from voiced and non-voiced boundaries; for each chunk, determine if the speaker for the chunk is enrolled as a speaker in a speaker database; when the speaker for the chunk is enrolled as a speaker in the speaker database, decode the chunk using a speaker-dependent automatic speech recognition (ASR) model that is specific for the speaker and tag the chunk with a permanent name for the speaker; when the speaker for the chunk is not enrolled as a speaker in the speaker database: assign a temporary name for the speaker of the chunk; tag the chunk with the temporary name; decode the chunk using a speaker independent speech recognition model; accumulate a predetermined number of chunks of audio that are tagged with the temporary name; and when the predetermined number of chunks is reached, enroll the speaker with the temporary name and the predetermined number of accumulated chunks in the speaker database and generate a speaker dependent ASR model using the predetermined number of accumulated chunks; format the decoded chunk as text using natural language processing; and display the formatted text on the graphical display unit along with the identity of the speaker for the formatted text, the identity comprising the permanent name of the speaker or the temporary name assigned to the speaker. The system is further configured to enroll a non-enrolled speaker into the speaker database and create a speaker-dependent ASR model for the non-enrolled speaker after a predetermined number of chunks of audio from the non-enrolled speaker are received.

The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in software executed by a processor, or in a combination of the two. A software may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

The process steps may be interchanged in any order without departing from the scope of the invention as long as falling within the appended claims.

Claim 1:
A flight deck system for extracting speaker information in an Air Traffic Controller, ATC, conversation and displaying the speaker information on a graphical display unit, the flight deck system comprising a controller configured to:
segment a stream of audio received over radio from an ATC and other aircraft into a plurality of chunks, wherein each chunk has a speaker;
for each chunk, determine if the speaker for the chunk is enrolled as a speaker in a speaker database;
when the speaker for the chunk is enrolled as a speaker in the speaker database, decode the chunk using a speaker-dependent automatic speech recognition, ASR, model that is specific for the speaker and tag the chunk with a permanent name for the speaker;
when the speaker for the chunk is not enrolled as a speaker in the speaker database, assign a temporary name for the speaker of the chunk, tag the chunk with the temporary name, and decode the chunk using a speaker independent speech recognition model;
format the decoded chunk as text;
signal the graphical display unit to display the formatted text along with an identity for the speaker of the formatted text, the identity comprising the permanent name of the speaker, or the temporary name assigned to the speaker; and
enroll a non-enrolled speaker into the speaker database and create a speaker-dependent ASR model for the non-enrolled speaker after a predetermined number of chunks of audio from the non-enrolled speaker are received, wherein
when the speaker for a chunk is not enrolled as a speaker in the speaker database, the controller is further configured to:
accumulate a predetermined number of chunks of audio that are tagged with a same temporary name; and
when the predetermined number of chunks is reached,
generate a speaker dependent ASR model for the speaker with the temporary name using the predetermined number of accumulated chunks; and
enroll the speaker with the temporary name and the speaker dependent ASR model for the speaker with the temporary name in the speaker database.