Patent Description:
In air traffic control, the controller communicates with the pilot through land-air voice calls, and the controller issues control instructions to the pilot, instructing the pilot to operate the aircraft as required. The role of the controller is critical. Many important decisions in the aircraft navigation are made by the controller based on the content of the dialogue between him/her and the pilot. Due to the professionalism, regional differences and personnel complexity of air traffic control, in addition to noisy environments, communication link interference and other factors, there are a large quantity of professional terms, unique regional names, mixed Chinese and English, and accent differences in the air traffic control speech. During the actual operation, a minor mistake such as mishearing or omission of information is quite likely to cause the pilot to make incorrect decisions based on the controller's speech, resulting in huge losses in the aviation field. Therefore, the pilot on the plane urgently needs an auxiliary means in the process of communicating with the controller.

With the development of natural language processing technology, speech recognition has been proposed as a technical approach to assist air traffic control information. However, it is difficult to achieve accurate and efficient speech recognition in the air traffic control system, which is mainly reflected in some aspects as follows: First, the environment in which the air traffic control system is located is special, which directly determines the particularity of the air traffic control speech recognition in acquiring speech information. Next, pronunciation of numbers, letters, flight numbers, runways, and the like are specifically defined in the air traffic control system, and the structure and sequence of dialogue sentences in the air traffic control system need follow specified rules. As a result, ordinary speech recognition products cannot be applied to air traffic control dialogue recognition. Furthermore, because minor speech recognition errors in the air traffic control system may also cause huge losses, the air traffic control speech recognition system has strict requirements on the accuracy of recognition.

There are some speech recognition methods dedicated to air traffic control in the related art, but generally the accuracy is not high enough, and therefore they are mostly used in ground scenarios, to help ground-to-air communication data analysis operations such as control command quality assessment, post-event analysis, and workload assessment. Therefore, a method and a system are urgently needed to assist pilots on board, improve the convenience and accuracy of air traffic control communication, and provide safety guarantees for aircraft navigation.

The patent publication no. <CIT> discloses a speech transmission system for reducing accent-induced anomalies in air-to-ground, ground-to-air, and air-to-air communications. In the system in question, a first audio stream is comprised of a plurality of words, upon which a plurality of independent voice-to-text conversions are performed. If it is determined that at least one of the plurality of independent voice-to-text conversions is error free, a text-to-voice conversion of the at least one error-free voice-to-text conversion is performed to create a second audio stream.

The patent publication no. <CIT> discloses a method, system, and computer program product for providing an indication that a received vehicle operation instruction can be performed. During operation of a vehicle a vehicle operation instruction is received and at least one vehicle performance parameter to perform the vehicle operation instruction is calculated. Then, a determination is made as to whether the calculated at least one vehicle performance parameter exceeds performance limitations of the vehicle. If at least one performance parameter exceeds a performance limitation, then a first alert is generated and output. The system can include a speech-to-text module that can receive spoken instructions from the radio transceiver and convert the spoken instructions to text. In various aspects, the speech-to-text module can also convert a text-based automated response generated by the system into a spoken response that can be transmitted via the radio transceiver to an air traffic controller.

The patent publication no. <CIT> discloses methods and systems that are provided for processing speech inputs for a controlling one or more vehicle systems of a vehicle. The method includes receiving speech input from an audio channel; performing, by a processor, speech recognition on the speech input to obtain recognized results; determining, by a processor, an accuracy level of the audio channel based on a comparison of the recognized results and predictive phraseology; determining, by a processor, an integrity level of the audio channel based on situational awareness information; communicating the recognized results, accuracy level, and the integrity level to a vehicle system; and selectively using the recognized results by the vehicle system based on the accuracy level and the integrity level. <CIT> discloses methods and systems for processing speech inputs for controlling one or more vehicle systems of a vehicle. Various aspects of speech recognition and/or processing are also disclosed in <CIT>, <NPL>; [<NPL>, <CIT>, <CIT>, <CIT>, and <CIT>. Document <CIT> discloses a device for aiding communication in the aeronautical domain, including a transceiver and data processor assembly that records audio messages corresponding to all the incoming and outgoing audio communications, transcribes the messages into textual messages in real time, displays the textual messages, and enables an audio playback of the audio messages.

The speech recognition methods dedicated to air traffic control in the above-mentioned documents still have the problems of low accuracy and difficulties associated with the analysis of speech data. Therefore, the need for a method and system that provide high accuracy and analysis still remains.

At present, there is no solution proposed to resolve the foregoing problem.

The present invention provides an on-board information assistance system and method, which is used to recognize, process, and use air recorded traffic control speech information according to the appended claims.

To describe the technical solutions in the embodiments of the present invention more clearly, the accompanying drawings required for the embodiments are briefly described below. Apparently, the accompanying drawings in the following description show merely some embodiments of the present invention, and a person of ordinary skill in the art may still derive other accompanying drawings from these accompanying drawings without creative efforts.

To enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention are clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, but not all of embodiments of the present invention. All other embodiments obtained by a person of ordinary skill in the art on the basis of the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

Terms "first" and "second" in the specification and claims of the present invention and the foregoing accompanying drawings are used for distinguishing similar objects and not necessarily used for describing a specific sequence. It should be understood that the data used in such a way may be exchanged under proper conditions to make it possible to implement the described embodiments of the present invention in other sequences apart from those illustrated or described here. In addition, terms "comprise", "include" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, a method, a system, a product or a device that includes a series of steps or units is not necessarily limited to the clearly listed steps or units, but may include other steps or units that are not clearly listed or are inherent to the process, the method, the product or the device.

According to an embodiment of the present invention, a method embodiment of an on-board information assistance method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be performed in a computer system based on a set of computer-executable instructions. In addition, although a logical order is shown in the flowchart, in some cases, steps shown or described may be performed in an order different from that herein.

<FIG> is a flowchart of an on-board information assistance system according to an embodiment of the present invention. As shown in <FIG>, the system includes the following modules:.

An obtaining module (<NUM>) is installed in an airborne communication system, and configured to obtain speech information, where the speech information is real-time air-ground communication speech information.

The speech information is acquired from an air-ground communication link. Specifically, to capture audio and speech transmission data between land and air in real time in the embodiments of the present invention, it is necessary to establish a dedicated speech communication link, and set up a speech capture program to record and intercept the speech data transmitted in the speech communication link, and transmit all the speech data to a processor for subsequent speech analysis and processing. The speech information may be acquired by using a land control device or an airborne speech control device. The control device for speech capture and transmission is not specifically limited thereto.

The speech information may be acquired by using a speech communication link or real-time communication data. Because the real time is usually important when the land and air pilots communicate with each other. The real-time voice calls can be kept and updated constantly on land and in the air to achieve the safety effect. Therefore, the speech needs to captured in real time according to the data in the communication link. An increasingly small capture latency causes an increasingly small impact on the coordinated control of the speech over land and air.

A speech processing module <NUM> is installed in the airborne information system and a speech processing module <NUM> is installed in an integrated modular avionics system. The speech processing modules are configured to convert the speech information into text information.

The speech processing modules convert the speech information into text information in the following manners:.

The speech information entropy is obtained according to the speech information. The speech information entropy represents the complexity of the speech information. In this embodiment of the present invention, after the complexity of the speech data is obtained, how to segment speech data is automatically determined according to the complexity and the length of speech data, and therefore, speech recognition can be performed more effectively to convert segmented speech data into word information. The sentences in the speech information are segmented according to the speech information entropy, and the segmented speech data is outputted as the single piece of speech data.

S23: Perform speech recognition according to the single piece of speech data to generate text data.

The speech processing module <NUM> in the airborne information system outputs first text information; and the first text information is outputted to an electronic flight bag <NUM> and is finally displayed on a man-machine interactive interface <NUM>.

The speech processing module <NUM> in the integrated modular avionics system outputs second text information; and the second text information is outputted to an instruction recording system <NUM> and is finally displayed on a man-machine interactive interface <NUM>.

The speech processing module <NUM> in the airborne information system and the speech processing module <NUM> in the integrated modular avionics system use heterogeneous speech information processing modules. The two modules back up each other, replacing the process of mutual confirmation of flight crews, reducing workload and improving flight efficiency.

The displaying effect of the man-machine interactive interface <NUM> is shown in <FIG>, and the displaying process specifically includes:
To recognize the segmented single piece of speech data and convert the recognized data into text data corresponding to the content of the single piece of speech data, it is necessary to recognize each frame of speech data in the single piece of speech data by using a speech recognition algorithm, and then recognized Chinese characters or characters are spliced and merged to obtain a complete text sentence and output it.

For example, if the played content of the single piece of speech data is "there is a thunderstorm in the air, land as soon as possible", the text content recognized by using the speech recognition algorithm includes "there is a thunderstorm in the air", ",", "land as soon as possible". Therefore, in the end, through the speech-text splicing operation, the foregoing recognized text content is spliced to obtain the text data of "there is a thunderstorm in the air, land as soon as possible", and then the text data is showed to the pilot, prompting the pilot to land as soon as possible.

Optionally, the step of displaying the single piece of speech data and the text data includes: playing the single piece of speech data; and displaying corresponding text data according to the played single piece of speech data.

Specifically, after the speech data is converted into text data, according to the requirements of pilots and ground control personnel, in the embodiment of the present invention, the single piece of speech data and text data are displayed simultaneously, that is, the corresponding text data is displayed while the speech data is played, such that the user can intuitively understand the content of the speech without being affected by the other party's accent and other factors. <FIG> is a schematic effect diagram of an on-board air traffic control information assistance system according to an embodiment of the present invention. As shown in <FIG>, when taping/clicking to play a speech, the user can see the displayed text below the speech, increasing the accuracy of speech information obtained by the user.

Because some important speech data needs to be played more than once to obtain complete speech information or commands, each single piece of speech data can be played at least once, that is, played multiple times according to user' requirements to completely understand the speech content or execute commands in the speech.

The convenience and accuracy can be increased when the control instruction is obtained in the foregoing steps.

A navigation system <NUM> receives a navigation instruction in the text information for flight guidance; and the navigation instruction is recorded in the instruction recording system <NUM>.

The navigation instruction information is extracted from the text information, and converted into an instruction format acceptable by the aircraft navigation system for flight guidance. The navigation instruction is further recorded in the instruction recording system.

<FIG> is a flowchart of an on-board information assistance method according to an embodiment of the present invention. The method corresponds to the on-board information assistance system in Embodiment <NUM>, and specifically includes:
S10: Obtain target speech information.

S20: Perform speech recognition on the target speech information by using at least two methods, to generate text information.

In this embodiment, two heterogeneous computing and processing modules installed respectively in the airborne information system and the integrated modular avionics system are used for speech recognition. The methods implemented in the two modules back up each other, replacing the process of mutual confirmation of flight crews, reducing the workload, and improving flight efficiency.

In this embodiment, one of the used speech recognition methods is as follows:.

Specifically, after the speech information is converted into text information, according to the requirements of pilots and ground control personnel, in the embodiment of the present invention, the single piece of speech data and text data are displayed simultaneously, that is, the corresponding text data is displayed while the speech data is played, such that the user can intuitively understand the content of the speech without being affected by the other party's accent and other factors. <FIG> is a schematic effect diagram of an on-board air traffic control information assistance system according to an embodiment of the present invention. As shown in <FIG>, when taping/clicking to play a speech, the user can see the displayed text below the speech, increasing the accuracy of speech information obtained by the user.

The method further includes S40: Extract navigation instruction information from the text information, and input the navigation instruction information into an aircraft navigation system for flight guidance, which specifically includes:
The navigation instruction information is extracted from the text information, and converted into an instruction format acceptable by the aircraft navigation system for flight guidance. The navigation instruction is further recorded in the instruction recording system.

In some embodiments provided by the present application, it should be understood that the disclosed technical content may be implemented in other manners. The described apparatus embodiment is merely an example. For example, the unit division may be a logical function division, and there may be other division manners in actual implementation. For example, a plurality of units and components may be combined or integrated into another system, or some features may be ignored or may not be performed. The indirect couplings or communication connections between the units or modules may be implemented in electric, mechanical, or other forms.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one position, or may be distributed on a plurality of units. Some or all of the modules may be selected according to actual requirements to implement the solution of the embodiment.

In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each of the modules may exist alone physically, or two or more modules are integrated into one unit. The foregoing integrated unit may be implemented in a form of hardware or software functional units.

Claim 1:
An on-board information assistance system, comprising:
an obtaining module (<NUM>), installed in an airborne communication system, and configured to obtain real-time air-ground communication speech information;
a speech processing module (<NUM>), installed in an airborne information system, configured to convert the speech information into text information,
a speech processing module (<NUM>) installed in an integrated modular avionics system, and configured to convert the speech information into text information,
the speech processing module (<NUM>) in the airborne information system is configured to output text information; that is displayed on a man-machine interactive interface (<NUM>);
wherein the speech processing module (<NUM>) in the integrated modular avionics system is configured to output text information, which is outputted to an instruction recording system (<NUM>) and is finally displayed on the man-machine interactive interface (<NUM>); and
a navigation system (<NUM>) is configured to receive a navigation instruction extracted from the text information for flight guidance,
the instruction recording system (<NUM>) is configured to record the navigation instruction,
wherein one of the speech processing modules (<NUM>, <NUM>) is configured to convert the speech information into text information in the following manners:
segmenting speech data according to information entropy of the speech data to obtain a single piece of speech data; and
performing speech recognition according to the single piece of speech data to generate text data and
the speech processing module (<NUM>) in the airborne information system and the speech processing module (<NUM>) in the integrated modular avionics system are two heterogeneous computing and processing modules,
wherein the man-machine interactive interface (<NUM>) is configured to simultaneously play the single piece of speech data and to display corresponding text data according to the single piece of speech data,
wherein the man-machine interactive interface (<NUM>) is configured such that a representation of the single piece of speech is displayed together with the corresponding text data, and a user can tap or click to play the speech, and the recognized text is displayed below the representation of the speech,
wherein the man-machine interactive interface (<NUM>) is configured such that each single piece of speech data can be played multiple times according to a user's requirements.