Translation device and system with utterance reinput request notification

A translation device is configured to acquire utterance spoken by a speaker in a first language and translate contents of the utterance into a second language for information presentation, and includes an input unit, a controller, a notification unit, and a storage. The input unit acquires the utterance in the first language and generates voice data from the utterance. The controller acquires a first evaluation value. The notification unit presents the speaker with information on utterance reinput request. The notification unit presents first information on utterance reinput request when the first evaluation value is less than or equal to a first predetermined value. The controller generates new voice recognition data with reference to the past voice recognition data and voice recognition data of reinput utterance, when the voice recognition data of the reinput utterance has an evaluation value less than or equal to a predetermined value.

TECHNICAL FIELD

The present disclosure relates to a translation device configured to translate utterance acquired in a language into another language.

BACKGROUND ART

PTL 1 discloses a translation system configured to translate by means of an automatic translation function of a computing machine. This translation system includes a translation converter configured to translate a language input via a language input unit by a first person into a translation language for a second person, a retranslation converter configured to retranslate the input language into a translation language for the first person, and a feedback language output unit configured to constantly present the first person with the retranslation language. The first person can thus check whether or not the translation language for the second person constantly has correct contents, and reinput different expression preferred for interpretation of the input language when the expressed contents are unintended.

CITATION LIST

Patent Literature

SUMMARY

The present disclosure provides a translation device configured to acquire utterance in a first language by a speaker, translate contents of the utterance into a second language for information presentation, and request the speaker to reinput when voice recognition processing or translation processing has an inappropriate result.

A translation device according to an aspect of the present disclosure is configured to acquire utterance spoken by a speaker in a first language and translate contents of the utterance into a second language for information presentation. The translation device includes an input unit, a controller, a notification unit, and a storage. The input unit acquires the utterance in the first language and generates voice data from the utterance. The controller acquires a first evaluation value for voice recognition data obtained through voice recognition processing of the voice data. The notification unit presents the speaker with information on utterance reinput request. The storage stores the voice recognition data as past voice recognition data. The notification unit presents first information on utterance reinput request when the first evaluation value is less than or equal to a first predetermined value. The controller generates new voice recognition data with reference to the past voice recognition data and voice recognition data of reinput utterance, when the voice recognition data of the reinput utterance has an evaluation value less than or equal to a predetermined value.

The translation device according to the present disclosure achieves requesting the speaker to reinput when voice recognition processing or translation processing has an inappropriate result. The translation device can present the speaker with information having contents appropriate for a processing result.

DESCRIPTION OF EMBODIMENTS

Exemplary embodiments will now be described in detail below with reference to the drawings where appropriate. The following description may not include unnecessarily detailed description. For example, the following description may not include detailed description of a well-known matter and repeated description of substantially identical configurations. This is to prevent unnecessary redundancy in the following description and to facilitate comprehension by those skilled in the art.

The inventors provide the accompanying drawings and the following description to help those skilled in the art to fully comprehend the present disclosure, but do not intend to limit subject matters recited in the claims with the drawings and the description.

First Exemplary Embodiment

A first exemplary embodiment will be described below with reference toFIGS. 1 to 5. Described below is a translation device including a voice input device and adopting a voice input method according to the present disclosure.

FIG. 1is an outer appearance view of the translation device according to the first exemplary embodiment. Translation device1depicted inFIG. 1is a tablet device or the like, and is configured to translate conversation between two users speaking different languages. The present exemplary embodiment assumes that translation device1translates face-to-face conversation between a guest (traveler) speaking English and a host (guide) speaking Japanese and guiding the guest.

Translation device1includes microphone10, speaker unit12, display14, and touch panel16. Microphone10and speaker unit12can be disposed near openings in a side surface of translation device1. Display14and touch panel16are disposed on a primary surface of translation device1. Display14has an area near a first end in a longitudinal direction (e.g., adjacent to the host), and the area includes utterance icons14h,14hgand display area15h. Display14has an area near a second end in the longitudinal direction (e.g., adjacent to the guest), and the area includes utterance icon14gand display area15g. Utterance icons14h,14g,14hgaccept touch operation by the users. The touch operation according to the present exemplary embodiment includes operation of touching and leaving an area corresponding to each of utterance icons14h,14g,14hgon touch panel16with a finger of the host or the guest, as well as operation of touching, then sliding, and leaving this area with the finger of the host or the guest.

Utterance icon14his an operation icon to be operated by the host to specify start and end points of own utterance when the host gives the utterance (i.e., upon inputting utterance in Japanese to translation device1). Utterance icon14gis an operation icon to be operated by the guest to specify start and end points of own utterance when the guest gives the utterance (i.e., upon inputting utterance in English). Utterance icon14hgis an operation icon to be operated by the host in place of the guest to specify start and end points of utterance by the guest when the guest gives the utterance (e.g., upon inputting utterance in English). Display areas15h,15gare provided for display of a result of voice recognition, translation, reverse translation, or the like in the form of a character string.

FIG. 2is a block diagram depicting an electric configuration of translation device1according to the first exemplary embodiment. Translation device1executes data communication with voice recognition server3, translation server4, voice synthesis server5, and evaluation server6via network2like the Internet.

Voice recognition server3receives digital voice data from translation device1via network2and executes voice recognition of the digital voice data thus received to generate voice recognition data in the form of a character string.

Translation server4receives the voice recognition data from translation device1via network2and translates the voice recognition data thus received to generate translation data in the form of a character string.

Voice synthesis server5receives the translation data in the form of a character string from translation device1via network2and executes voice synthesis of the translation data in the form of a character string thus received to generate a voice signal.

Evaluation server6receives the voice recognition data or the translation data from translation device1via network2and calculates an evaluation value indicating a level of “sentence perfection” of a sentence according to the voice recognition data or the translation data. Such “sentence perfection” indicates appropriateness as a sentence in the language of the sentence.

Microphone10is a device configured to convert voice into digital voice data. Specifically, microphone10converts voice into a voice signal (analog electric signal), and further converts the voice signal into digital voice data with use of an AD converter. In other words, microphone10acquires utterance by a speaker and generates voice data according to the utterance.

Communication unit18is a communication module configured to execute data communication with voice recognition server3, translation server4, voice synthesis server5, and evaluation server6via network2in accordance with a communication system such as Bluetooth (registered trademark), Wi-Fi (registered trademark), 3G, long term evolution (LTE) (registered trademark), or the Institute of Electrical and Electronics Engineers (IEEE) 802.11.

Storage20is a recording medium including a flash memory, a ferroelectric memory, a hard disk drive (HDD), a solid state drive (SSD), or the like. Storage20stores digital voice data from microphone10and translation data from translation server4. Storage20also stores various programs for controller22.

Controller22includes a central processing unit (CPU), a micro processing unit (MPU), or the like and executes the various programs stored in storage20to control entire behavior of translation device1. Controller22according to the present exemplary embodiment has a function that is achieved through cooperation between hardware and software, and can alternatively be achieved only with a hardware circuit dedicatedly designed for achievement of a predetermined function. Controller22is not limited to the CPU or the MPU, but can include a digital signal processor (DSP), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or the like.

Speaker unit12is configured to convert an electric signal into voice. Speaker unit12outputs voice according to the voice signal (electric signal) from controller22.

Display14is configured to display an image, and is configured by a liquid crystal display device or an organic electroluminescence (EL) display device. Display14displays, in display area15h,15g, an image according to voice recognition data, translation data, or reverse translation data from controller22. Display14exemplifies a notification unit configured to present the host and the guest with voice recognition data, translation data, and reverse translation data. Display14also displays utterance icons14h,14g,14hgdescribed above.

Touch panel16is an operation unit to be operated by the user and accepts a command from the user. Touch panel16is superimposed on display14.

Translation device1configured as described above will be described in terms of behavior.

Translation device1transmits, to voice recognition server3via network2, digital voice data according to voice input to microphone10during a voice input period. Voice recognition server3executes voice recognition of the voice data thus received to generate voice recognition data (text data) in the form of a character string. Translation device1receives the voice recognition data from voice recognition server3via network2.

Translation device1transmits the voice recognition data to translation server4via network2. Translation server4translates the voice recognition data thus received to generate translation data (text data) in the form of a character string. Translation device1receives the translation data from translation server4via network2.

Translation device1transmits the translation data to voice synthesis server5via network2. Voice synthesis server5synthesizes voice in accordance with the translation data thus received to generate a voice signal. Translation device1then receives the voice signal from voice synthesis server5via network2.

Translation device1outputs, from speaker unit12, voice expressing a translation result in accordance with the voice signal thus received. Translation device1simultaneously presents on display14text information (the translation result) according to the translation data.

When translation device1translates utterance by the host, translation device1initially determines a voice input period for voice recognition in accordance with touch operation by the host to utterance icon14hon touch panel16. Specifically, translation device1determines a start point of the voice input period when the host firstly touches utterance icon14h, and determines an end point of the voice input period when the host secondly touches utterance icon14h. Translation device1executes voice recognition and translation of voice of the host input to microphone10during the voice input period from the start point to the end point thus determined. The end point of utterance can alternatively be set after elapse of predetermined time from the start point, in consideration of possible failure in touching the operation icon or redundant voice input. Translation device1outputs the voice expressing the translation result from speaker unit12. Translation device1simultaneously presents the translation result in the form of a character string in display area15gadjacent to the guest, of display14, and presents the voice recognition result (as well as the reverse translation result as necessary) in the form of a character string in display area15hadjacent to the host, of display14.

When translating utterance by the guest, translation device1initially determines a voice input period in accordance with touch operation by the guest to utterance icon14gon touch panel16. Specifically, translation device1determines a start point of the voice input period when the guest firstly touches utterance icon14g, and determines an end point of the voice input period when the guest secondly touches utterance icon14g. Translation device1executes voice recognition and translation of voice of the guest input to microphone10during the voice period from the start point to the end point thus determined. Similarly to the above case, the end point of utterance can alternatively be set after elapse of predetermined time from the start point, in consideration of possible failure in touching the operation icon or redundant voice input. Translation device1outputs, from speaker unit12, the translation result in the form of voice. Translation device1simultaneously presents the translation result in the form of a character string in display area15hadjacent to the host, of display14, and presents the voice recognition result (as well as the reverse translation result as necessary) in the form of a character string in display area15gadjacent to the guest, of display14.

Translation device1thus configured will not eventually obtain a correct translation result if halfway voice recognition processing or translation processing does not have a result of appropriate language expression. With such an eventually inappropriate translation result, the user needs extra time to reinput utterance (voice). In an exemplary case where voice recognition processing has an inappropriate result, subsequent translation processing, voice synthesis processing, and the like need to be executed even though no correct result may be obtained eventually. In another case where translation processing has an inappropriate result, subsequent voice synthesis processing and the like also need to be executed. Such eventually useless processing needs time to be executed.

In view of this, the present exemplary embodiment includes, instead of executing subsequent processing, requesting the user to reinput utterance (voice) when voice recognition processing or translation processing has an inappropriate result and such inappropriateness is detected. In an exemplary case where voice recognition of utterance by the host has a determination result of inappropriateness as a sentence in Japanese, subsequent processing is not executed and display area15hadjacent to the host has a message requesting reinput of utterance, as depicted inFIG. 3. This eliminates useless processing according to inappropriate text information and enables promptly requesting the user to reinput utterance (voice).

FIG. 4is a table exemplifying messages displayed upon utterance reinput request. When voice recognition has a low evaluation result, there is displayed a message “Please speak clearly again” (exemplifying first information). When translation processing has a low evaluation result, there is displayed a message “Please speak again in different expression” (exemplifying second information). When reverse translation (to be described later) has a low evaluation result, there is displayed a message “Please check whether or not your partner receives what you want to tell” (exemplifying fourth information). The message requesting reinput is not displayed when voice recognition, translation processing, and reverse translation processing each have high evaluation. In this manner, the displayed message varies in accordance with the type of processing having low evaluation. This achieves more accurate notification to the user of what to focus upon reinput of utterance, to reduce a risk of third input of utterance. The first exemplary embodiment does not include evaluation of reverse translation.

FIG. 5is a flowchart of translation processing executed by controller22of translation device1according to the first exemplary embodiment. The translation processing executed by translation device1will be described below with reference to the flowchart inFIG. 5. The following description assumes a case where translation device1translates utterance (voice) in Japanese by the host (e.g., a guide) into English and a result of the translation is presented to the guest (e.g., a traveler).

Microphone10acquires utterance (voice) of the user and generates voice data (S11). Controller22acquires the voice data from microphone10and executes voice recognition to generate voice recognition data in the form of a character string (S12). Specifically, controller22transmits the voice data to voice recognition server3via communication unit18. Voice recognition server3executes voice recognition in accordance with the voice data thus received, generates voice recognition data, and transmits the voice recognition data thus generated to translation device1.

Controller22subsequently evaluates the voice recognition data thus received (S13). Specifically, controller22transmits the voice recognition data thus received to evaluation server6via communication unit18. Evaluation server6calculates, from text (hereinafter, referred to as “voice recognition text”) obtained from the voice recognition data thus received, a first evaluation value indicating a level of “sentence perfection” of a Japanese sentence according to the voice recognition text (S13).

Such “sentence perfection” indicates appropriateness as to whether or not the sentence appears naturally. The level of “sentence perfection” is calculated in accordance with appearance probability of each word configuring the sentence. Specifically, the level of “sentence perfection” relates to evaluation of a first word according to appearance probability of the first word in positional relation with a second word adjacent to the first word. The appearance probability of the first word is preliminarily calculated through analysis of mass sentence data. An N-gram model (a bi-gram model (N=2) in the present exemplary embodiment) exemplifies a technique of evaluating sentence perfection in accordance with such appearance probability. Evaluation server6stores information (a table) that is generated by preliminary data analysis according to the N-gram model and correlates a first word with appearance probability of the first word before or after a second word adjacent to the first word.

Assume an exemplary case of obtaining voice recognition text of sentence A10depicted inFIG. 6.FIG. 6exemplifies a case where word A2appears subsequently to word A1with probability of 0.1, word A3appears subsequently to word A2with probability of 0.0001, word A4appears subsequently to word A3with probability of 0.2, word A5appears subsequently to word A4with probability of 0.15, and word A6appears subsequently to word A5with probability of 0.3. Evaluation server6obtains, as the first evaluation value for sentence perfection, a geometrical mean of appearance probability values of the respective words. Sentence A10has the first evaluation value obtained as a fifth root of (0.1×0.0001×0.2×0.15×0.3). The first evaluation value is set to be higher as “sentence perfection” has a higher level in this example.

Evaluation server6transmits the first evaluation value thus obtained to translation device1. Controller22of translation device1compares the first evaluation value with a first predetermined value (S14).

In a case where the first evaluation value is less than or equal to the first predetermined value (NO in S14), controller22sets a message (first information) requesting the host to reinput utterance (voice) (S20). Controller22also sets a message to the guest. Controller22then presents in display area15hadjacent to the host, of display14, the message requesting the host to reinput (S21). Controller22simultaneously presents the message to the guest also in display area15gadjacent to the guest. This completes the translation processing.

FIG. 3exemplifies a case where display area15hhas a message “Please speak clearly again (in Japanese)” to the host and display area15ghas a message “Please wait. Re-speech is being requested.” to the guest. These messages are displayed to allow the host to recognize necessity of reinput of voice (utterance) as well as allow the guest to recognize necessity of waiting for a while for reinput of voice. The message “Please speak clearly (in Japanese)” also allows the host to recognize that own voice (utterance) has been unclear and next utterance needs to have clearer pronunciation.

In another case where the first evaluation value exceeds the first predetermined value (YES in S14), controller22executes translation processing according to the voice recognition data (voice recognition text) (S15). Specifically, controller22transmits the voice recognition data to translation server4via communication unit18. Translation server4translates the voice recognition data thus received and transmits, to translation device1, translation data including text according to a translation result.

Controller22of translation device1having received the translation data (text data) evaluates the translation data thus received (S16). Specifically, controller22transmits the translation data thus received to evaluation server6via communication unit18. Evaluation server6calculates, from text (hereinafter, referred to as “translation text”) obtained from the translation data thus received, a second evaluation value indicating a level of “sentence perfection” of an English sentence according to the translation text (S16), and transmits the second evaluation value thus calculated to translation device1.

Controller22of translation device1compares the second evaluation value with a second predetermined value (S17).

In a case where the second evaluation value is less than or equal to the second predetermined value (NO in S17), controller22sets a message (second information) requesting the host to reinput utterance (voice) (S20). Controller22also sets a message to the guest. The set message is different in terms of contents from the message requesting reinput when the first evaluation value for a voice recognition result is low, as depicted inFIG. 4. In an exemplary case where the first evaluation value for the voice recognition result is low, there is displayed a message “Please speak clearly again”. In another case where the second evaluation value for the translation result is low, there is displayed a message “Please speak again in different expression”. Different messages are displayed in accordance with different reinput reasons in these manners, to allow the user to recognize a reinput reason and be requested to reinput more appropriate utterance.

Controller22then displays in display area15hadjacent to the host, of display14, a message requesting the host to reinput (S21). Controller22simultaneously presents the message to the guest also in display area15gadjacent to the guest. This completes the translation processing.

In another case where the second evaluation value exceeds the second predetermined value (YES in S17), controller22transmits the translation data to voice synthesis server5for execution of voice synthesis processing (S18). Voice synthesis server5executes voice synthesis in accordance with the translation data thus received, and transmits, to translation device1, voice data for generation of voice expressing the translation result.

Controller22of translation device1outputs, from speaker unit12, voice according to the voice data received from voice synthesis server5(S19). Controller22simultaneously presents, in display area15hof display14, a sentence according to the translation data (S19).

As described above, the utterance by the host is translated and the translation result is presented to the guest in the form of voice and character information. In particular, translation device1according to the present exemplary embodiment displays a message requesting the user to reinput without executing subsequent processing, when each of voice recognition processing and translation processing has an evaluation result of inappropriateness as a sentence (lacking sentence perfection). This configuration eliminates execution of useless processing according to an inappropriate voice recognition or translation result and enables promptly requesting the user to reinput. The displayed message requesting the user to reinput differs in accordance with whether voice recognition or translation has a low evaluation result. This achieves display of an appropriate message according to a situation. The user can recognize how to reinput with reference to the message.

[1-3. Effects and Others]

As described above, translation device1according to the present exemplary embodiment is configured to acquire utterance in the first language (e.g., Japanese) by the speaker and translate contents of the utterance into the second language (e.g., English) for information presentation. Translation device1includes microphone10(exemplifying an input unit), controller22, and display14(exemplifying a notification unit). Microphone10acquires utterance in the first language and generates voice data according to the utterance. Controller22obtains the first evaluation value for voice recognition data obtained through voice recognition processing of the voice data, and the second evaluation value for translation data obtained through translation processing of the voice recognition data into the second language. Display14presents a first message requesting reinput of utterance when the first evaluation value is less than or equal to the first predetermined value (S14), and presents a second message requesting reinput of utterance and being different from the first message (S21) when the first evaluation value is more than the first predetermined value and the second evaluation value is less than or equal to the second predetermined value (S17).

Translation device1configured as described above displays a message requesting the speaker to reinput when each of voice recognition processing and translation processing has an evaluation result of inappropriateness as a sentence (lacking sentence perfection). This enables promptly requesting the speaker to reinput. The displayed message requesting the speaker to reinput differs in accordance with whether voice recognition or translation has a low evaluation result. This achieves display of an appropriate message according to a processing result. The speaker can recognize how to reinput with reference to the message.

Controller22executes neither translation processing (S15) nor voice synthesis processing (S18) subsequent to voice recognition processing, when the first evaluation value for a result of the voice recognition processing is found to be less than or equal to the first predetermined value. Furthermore, controller22does not execute subsequent voice synthesis processing (S18) when the second evaluation value for a result of translation processing is found to be less than or equal to the second predetermined value. This enables promptly requesting the speaker to reinput.

Second Exemplary Embodiment

Description is made to translation device1according to a different exemplary embodiment. Translation device1according to the present exemplary embodiment is configured to generate voice recognition data or translation data with reference to past data when voice recognition or translation of reinput utterance has a low evaluation result. Translation device1according to the present exemplary embodiment is similar in hardware configuration to the translation device according to the first exemplary embodiment.

FIG. 7is a flowchart of translation processing executed by translation device1according to the second exemplary embodiment. The flowchart inFIG. 7includes, in addition to steps S11to S21included in the flowchart inFIG. 5according to the first exemplary embodiment, steps S14-1to S14-3, S17-1to S17-4.

Processing from step S11to step S21according to the present exemplary embodiment is basically similar to that described in the first exemplary embodiment. Described below are differences from the processing included in the flowchart according to the first exemplary embodiment.

Controller22according to the present exemplary embodiment stores in storage20, voice data, voice recognition data, and translation data, upon acquisition of the voice data (S11), upon acquisition of the voice recognition data (S12), and upon acquisition of the translation data (S15), respectively. Controller22does not necessary store in storage20all of the voice data, the voice recognition data, and the translation data. Controller22can alternatively store only the voice data and generate voice recognition data and translation data from the stored voice data as necessary. Controller22can still alternatively store in storage20only the voice recognition data and the translation data, without storing the voice data.

If the first evaluation value for a voice recognition result is less than or equal to the first predetermined value (NO in S14), controller22determines whether or not current utterance input corresponds to reinput request (S14-1).

In a case where the current utterance input does not correspond to reinput request (NO in S14-1), controller22sets a message requesting reinput (S20) and presents the message on display14(S21), as in the first exemplary embodiment.

In another case where the current utterance input corresponds to reinput request (YES in S14-1), controller22generates new voice recognition text with reference to a past voice recognition result (S14-2). Controller22generates the new voice recognition text with reference to current voice recognition text (voice recognition data of reinput utterance) and previous voice recognition text (past voice recognition data), for example. An exemplary case will be described with reference toFIG. 8.

FIG. 8exemplifies sentence B10as the previous (first) voice recognition text, and sentence B20as the current (second) voice recognition text. In this case, sentence B30as the new voice recognition text is generated in accordance with the previous voice recognition text and the current voice recognition text. Specifically, among words configuring the previous voice recognition text, a word having appearance probability less than a predetermined value is replaced with a word positioned correspondingly in the current voice recognition text. As exemplified inFIG. 8, word B1in the previous voice recognition text has appearance probability (0.001) less than the predetermined value (e.g., 0.005) and is thus replaced with word B2in the current voice recognition text to obtain sentence B30as new voice recognition text.

Controller22can alternatively generate new voice recognition data through selection of a word having higher appearance probability between the previous voice recognition text and the current voice recognition text. Specifically, controller22does not execute comparison with the predetermined value but compares appearance probability (0.001) of word B1in sentence B10with appearance probability (0.1) of word B2included in sentence B20and corresponding to word B1. Controller22can then generate sentence B30through selection of word B2having higher appearance probability.

With reference toFIG. 7again, controller22then evaluates the new voice recognition text (S14-3). The voice recognition text is evaluated in accordance with the method described earlier (steps S13, S14). In a case where the new voice recognition text has low evaluation (NO in S14-3), in other words, where the first evaluation value for the new voice recognition text is less than or equal to the first predetermined value, controller22sets a message requesting reinput (S20) and presents the message on display14(S21). In another case where the new voice recognition text has high evaluation (YES in S14-3), the flow proceeds to translation steps (S15, S16).

If the second evaluation value for the translation result is less than or equal to the second predetermined value (NO in S17), controller22determines whether or not current utterance input corresponds to reinput request (S17-1).

In a case where the current utterance input does not correspond to reinput request (NO in S17-1), controller22sets a message requesting reinput (S20) and presents the message on display14(S21), as in the first exemplary embodiment.

In another case where the current utterance input corresponds to reinput request (YES in S17-1), controller22generates new translation text with reference to a past translation result (S17-2). Controller22generates the new translation text with reference to current translation text and previous translation text, for example. An exemplary case will be described with reference toFIG. 9.

FIG. 9exemplifies the past (first) translation text “You can go to Tokyo by bath” and the current (second) translation text “To Tokyo you can go by bus”. In this case, there is generated new translation text “You can go to Tokyo by bus” in accordance with the previous translation text and the current translation text. Specifically, a word having appearance probability less than or equal to the predetermined value in the previous translation text is replaced with a word positioned correspondingly in the current voice recognition text. As exemplified inFIG. 9, “bath” in the previous translation text has appearance probability (0.0) less than the predetermined value (e.g., 0.005) and is thus replaced with “bus” in the current translation text to obtain new translation text.

Controller22can alternatively generate new translation data through selection of a word having higher appearance probability between the previous translation text and the current translation text. Specifically, controller22does not execute comparison with the predetermined value but compares appearance probability (0.0) of “bath” with appearance probability (0.02) of “bus”. Controller22can then generate new translation data through selection of “bus” having higher appearance probability.

With reference toFIG. 7again, controller22thereafter evaluates the new translation text (S17-3). The translation text is evaluated in accordance with the method described earlier (steps S16, S17). In a case where the new translation text has low evaluation (NO in S17-3), in other words, where the second evaluation value for the new translation text is less than or equal to the second predetermined value, controller22sets a message requesting reinput (S20) and presents the message on display14(S21). In another case where the new translation text has an evaluation value exceeding the predetermined value (YES in S17-3), controller22deletes past voice data, past voice recognition data, and past translation data stored in storage20(S17-4). In other words, storage20continuously stores the respective data of input utterance until new translation text achieves high evaluation. Storage20can thus continuously hold data necessary for generation of new translation text until new translation text achieves high evaluation.

As described above, the present exemplary embodiment includes generating new text to be processed with reference to past voice recognition data or past translation data when voice recognition or translation of reinput utterance has an unpreferred result. This lowers frequency of reinput and accordingly reduces time necessary for translation processing.

Controller22deletes voice recognition data from storage20in step S17-4described above. Controller22can alternatively delete past voice recognition data from storage20when new voice recognition data has an evaluation value exceeding the predetermined value in step S14-3.

Controller22of translation device1according to the present exemplary embodiment generates new voice recognition data or new translation data. The present disclosure is, however, not limited to this configuration. Evaluation server6can alternatively generate new voice recognition data or new translation data.

If NO in step S14-3, display14can alternatively present new voice recognition data in step S20, in addition to information on utterance reinput request. The speaker can accordingly recognize new voice recognition data upon reinput of utterance.

Third Exemplary Embodiment

Described below is a translation device according to a still different exemplary embodiment. The translation device according to any one of the exemplary embodiments described above sets information to be presented to the speaker in accordance with the first evaluation value for voice recognition data in the first language (Japanese) or the second evaluation value for translation data in the second language (English). Simple evaluation according to each language model hardly leads to sufficient evaluation of translation appropriateness. In view of this, translation device1according to the present exemplary embodiment sets information to be presented to the speaker in accordance with a third evaluation value for identity between voice recognition data and translation data. The third evaluation value is generated in accordance with distributed representation for the voice recognition data and the translation data. Translation device1according to the present exemplary embodiment is similar in hardware configuration to the translation device according to the first exemplary embodiment.

FIG. 10is a flowchart of translation processing executed by translation device1according to the third exemplary embodiment. The flowchart inFIG. 10includes, in addition to steps S11to S21included in the flowchart inFIG. 5according to the first exemplary embodiment, steps S17-11to S17-13.

Processing from step S11to step S21according to the present exemplary embodiment is basically similar to that described in the first exemplary embodiment. Described below are differences from the processing included in the flowchart according to the first exemplary embodiment.

Controller22of translation device1according to the present exemplary embodiment generates a first distributed representation group from voice recognition data in Japanese in accordance with a conversion table (first conversion table) for conversion of a Japanese word to distributed representation, if the second evaluation value exceeds the second predetermined value (YES in S17). Controller22further generates a second distributed representation group from translation data in English in accordance with a conversion table (second conversion table) for conversion of an English word to distributed representation (S17-11). These conversion tables may be prepared for conversion of not a word but a phrase or a sentence to distributed representation. These distributed representation groups will be described below with reference toFIGS. 11A and 11B.

FIG. 11Ais a view exemplifying voice recognition data in Japanese.FIG. 11Bis a view exemplifying translation data in English.FIG. 11Adepicts sentence C10according to the voice recognition data in Japanese, and sentence C10includes words C11to C14.FIG. 11Bsimilarly depicts sentence C20according to the translation data in English, and sentence C20includes words C21to C24.

Controller22converts each of words C11to C14to distributed representation in accordance with the first conversion table. Such distributed representation is exemplified herein by representation for a word, a phrase, or a sentence with a vector including a plurality of combined numbers. The distributed representation is exemplified below as a word vector through representing, with a vector, a word or a plurality of combined words assumed as a single word. The distributed representation for each of words C11to C14configures the first distributed representation group. Controller22calculates a sum of the distributed representation indicated by vectors and included in the first distributed representation group. Controller22then calculates sentence vector Sfof the first distributed representation group through division of the vectors as the sum of the distributed representation by four equal to a number of words. Assuming that the distributed representation for each of words C11to C14has vector Fiand the number of words is N (N=4 in this case), sentence vector Sfis expressed by expression (1) below.

Controller22similarly converts each of words C21to C24into distributed representation in accordance with the second conversion table. The distributed representation for each of words C21to C24configures the second distributed representation group. Controller22calculates a sum of the distributed representation indicated by vectors and included in the second distributed representation group. Controller22then calculates sentence vector Seof the second distributed representation group through division of the vectors as the sum of the distributed representation by four equal to the number of words. Assuming that the distributed representation for each of words C21to C24has vector Eiand the number of words is M (M=4 in this case), sentence vector Seof the second distributed representation group is expressed by expression (2) below.

The number (N) of words included in the first distributed representation group is equal to the number (M) of words included in the second distributed representation group in the present exemplary embodiment. It is possible to similarly calculate a sentence vector of each distributed representation group even when the number of words included in the first distributed representation group is different from the number of words included in the second distributed representation group.

The first conversion table and the second conversion table can alternatively be generated from a single parallel translation table (bilingual corpus). More specifically, the first conversion table may be generated from a Japanese part of the single parallel translation table, and the second conversion table may be generated from an English part of the parallel translation table. The conversion tables generated from the single parallel translation table achieve improvement in correspondence accuracy of distributed representation between the languages. This improves correspondence accuracy of the sentence vectors between the languages. This leads to improvement in accuracy of the third evaluation value calculated in accordance with the sentence vectors. A single parallel translation table can alternatively include two parallel translation tables substantially identical with each other. Improvement in accuracy of the third evaluation value is effected if the conversion tables are generated from two parallel translation tables including many common bilingual sentences.

Controller22generates the third evaluation value in accordance with sentence vector Sfand sentence vector Se(S17-12). Specifically, the third evaluation value (cosine similarity: cos θ) is calculated by expression (3) below. The third evaluation value is thus generated in accordance with identity between the first distributed representation group and the second distributed representation group.

Controller22compares the third evaluation value with a third predetermined value (S17-13). In a case where the third evaluation value is less than or equal to the third predetermined value (e.g., 0.8) (NO in S17-13), controller22sets a message (third information) requesting reinput of utterance (S20). As exemplified inFIG. 12, controller22sets a message requesting reinput of utterance, specifically, “Please speak again using different words”. Display14then presents the host (speaker) with the message (S21).

In another case where the third evaluation value exceeds the third predetermined value (YES in S17-13), controller22executes voice synthesis (S18), outputs voice according to the translation result from speaker unit12, and presents text according to the translation result in display areas15h,15gof display14(S19).

As depicted inFIG. 12, the third information is different from both the first information and the second information. In a case where display14presents the third information “Please speak again using different words”, the speaker can recognize that there is a problem not in voice recognition processing or translation processing but in identity between voice recognition data and translation data. That is, the speaker can recognize that he or she needs to use different words in utterance because own utterance has contents inappropriate for translation processing.

As described above, the present exemplary embodiment includes presenting a message requesting reinput and different from the first information and the second information, in accordance with the third evaluation value for identity between voice recognition data and translation data. This enables presenting the speaker with an appropriate message.

Controller22executes processing in step S20if the second evaluation value is less than or equal to the second predetermined value in step S17. Controller22can alternatively execute processing in step S17-11, regardless of the second evaluation value (skipping step S17). If the third evaluation value is less than or equal to the third predetermined value in step S17-13, controller22can alternatively set a message to be presented on display14in accordance with the second evaluation value and the third evaluation value (S20). Specifically, as depicted inFIG. 12, controller22may set a message “Please speak briefly again” as information on utterance reinput request if the second evaluation value is less than or equal to the second predetermined value and the third evaluation value is less than or equal to the third predetermined value. The speaker can thus recognize that there is a problem not in voice recognition processing but in translation processing and identity between voice recognition data and translation data. As described above, display14according to the present exemplary embodiment may present information on utterance reinput request, different from the first information, the second information, and the third information, if the second evaluation value is less than or equal to the second predetermined value and the third evaluation value is less than or equal to the third predetermined value.

The present exemplary embodiment adopts, as the third evaluation value, cosine similarity of respective sentence vectors. However, the present disclosure is not limited to this case. Examples of the third evaluation value include a Pearson's correlation coefficient and deviation pattern similarity.

Fourth Exemplary Embodiment

Translation device1according to the present exemplary embodiment has a reverse translation function of translating, into an original language (e.g., Japanese), a translation result (sentence) obtained by translation of a language of utterance (e.g., Japanese) into another language (e.g., English).FIG. 13exemplifies a reverse translation result presented along with a translation result on display14. Display area15hadjacent to the host as the speaker presents sentence D1according to a voice recognition result and sentence D2according to a reverse translation result. Display area15gadjacent to the guest presents a translation result “What are you looking for?”.

Translation device1according to the present exemplary embodiment evaluates the reverse translation result, and displays a message requesting reinput of utterance without outputting the translation result if the reverse translation result has low evaluation. Translation device1according to the present exemplary embodiment is similar in hardware configuration to the translation device according to the first exemplary embodiment.

FIG. 14is a flowchart of translation processing executed by translation device1according to the fourth exemplary embodiment. The flowchart inFIG. 14includes, in addition to steps S11to S21included in the flowchart inFIG. 5according to the first exemplary embodiment, steps S17-21to S17-23.

Controller22of translation device1according to the present exemplary embodiment evaluates the translation result (S16) and then executes reverse translation of the translation result (S17-21). Controller22accordingly transmits data of the translation result to translation server4. Translation server4executes reverse translation of text according the translation result thus received, and transmits, to translation device1, reverse translation data indicating a reverse translation result.

Controller22having received the reverse translation data obtains an evaluation value for the reverse translation result (S17-22). Controller22accordingly transmits the voice recognition data and the reverse translation data to evaluation server6. Evaluation server6calculates a fourth evaluation value for the reverse translation result from the voice recognition data and the reverse translation data. The fourth evaluation value for the reverse translation result is calculated in the following manner.

Specifically, the fourth evaluation value is calculated in accordance with sentence closeness (distance) between text according to data of the voice recognition result (hereinafter, referred to as “voice recognition text”) and text according to data of the reverse translation result (hereinafter, referred to as “reverse translation text”). Such sentence closeness can exemplarily be calculated by vectorizing a sentence according to the voice recognition text as well as a sentence according to the reverse translation text (see Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S. Zemel, Antonio Torralba, Raquel Urtasun, Sanja Fidler, “Skip-Thought Vecors”, arXiv: 1506.06726, 2015. 103) and obtaining cosine similarity between these sentence vectors. The sentence closeness can alternatively be obtained in accordance with calculated similarity or distance between words configuring the sentences. Specifically, similarity or distance may be obtained for each combination of words between the two sentences, and the sentence closeness may be obtained as a geometrical mean of all similarity or distance values thus obtained. The fourth evaluation value is calculated in accordance with the sentence closeness thus obtained. There is set an expression for calculation of the fourth evaluation value such that the fourth evaluation value is higher as the sentences are closer, in other words, as the similarity is larger or the distance is shorter.

The sentence closeness can be evaluated in accordance with a method examples of which include BLEU, BLEU+, WER, TER, RIBES, NIST score, METEOR, ROUGE-L, and IMPACT (see Graham Neubig, “Investigation of sentence-level machine translation evaluation measure”, Research report by Information Processing Society of Japan, 1, 2013; and Tsutomu Hirao, Hideki Isozaki, Kevin Duh, Katsuhito Sudo, Hajime Tsukada, and Masaaki Nagata, “RIBES: method of automatically evaluating translation according to rank correlation”, collected papers presented at 17th annual meeting of the Association for Natural Language Processing, 1115, 2011). Furthermore, a method of evaluating sentence closeness in consideration of sentence contents can be exemplified by a neural network method with a single hidden layer, a recurrent neural network method, a convolutional neural network method, a recursive neural network method, or a feedforward neural network method (see Yuta Tsuboi, “Development in deep learning in natural language processing”, Operations Research, 205, 2015). Vectorization of a word or a sentence also exemplifies the method of evaluating sentence closeness (see Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean, “Efficient Estimation of Word Representations in Vector Space”, arXiv: 1301.3781, 2013).

Evaluation server6transmits the fourth evaluation value thus calculated to translation device1. Translation device1compares the fourth evaluation value thus received with a fourth predetermined value (S17-23).

In a case where the fourth evaluation value is more than the fourth predetermined value (YES in S17-23), controller22executes voice synthesis (S18), outputs voice according to the translation result from speaker unit12, and presents text according to the translation result in display areas15h,15gof display14(S19).

In another case where the fourth evaluation value is less than or equal to the fourth predetermined value (NO in S17-23), controller22sets a message suggesting that translation may be inappropriate (S20). When the fourth evaluation value is less than or equal to the fourth predetermined value, the translation text is largely different in contents from the reverse translation text and the output translation result will be unlikely to be intended by the speaker. The message presented in the display area adjacent to the speaker can be set to include text “Please check whether or not your partner receives what you want to tell” as exemplified inFIG. 4. The message presented in the display area adjacent to the partner is set to ask for time. The message (fourth information) presented when the fourth evaluation value for the reverse translation result is low is different from the message presented when the evaluation value for the voice recognition result or the translation result is low, as depicted inFIG. 4. The contents of the message differ in accordance with the type of processing having low evaluation, to achieve presenting the speaker with an appropriate message according to a situation.

Controller22then presents the message thus set on display14(S21).FIG. 15is a view exemplifying presentation in this case. As depicted inFIG. 15, display area15hadjacent to the host presents text according to the voice recognition result, text according to the translation result “What are you waiting for?”, as well as text according to the reverse translation result and text of a message checking whether or not reinput is necessary. Display area15gadjacent to the guest presents text according to the translation result “What are you waiting for?” and a message indicating a risk of an incorrect translation result “The text shown above may be incorrect”. The host as the speaker determines whether or not utterance needs to be reinput through checking the contents presented in display area15h, and reinputs utterance if necessary. The host can recognize whether or not translation device1has correctly translated what the host wants to tell, and can select appropriate words to correctly input the contents of utterance.

As described above, the present exemplary embodiment further includes evaluating the reverse translation result and displaying a message requesting reinput if the reverse translation has low evaluation. This configuration prevents output of an inappropriate translation result.

The present exemplary embodiment includes addition of steps S17-21to S17-23to the steps in the flowchart inFIG. 5according to the first exemplary embodiment. The present exemplary embodiment can alternatively include addition of steps S17-21to S17-23to the steps in the flowchart inFIG. 10according to the third exemplary embodiment. In the latter case, processing in step S17-21may be executed if YES in step S17-13.

Other Exemplary Embodiments

The first to fourth exemplary embodiments have been described above to exemplify the techniques disclosed in the present application. The techniques according to the present disclosure are not limited by these exemplary embodiments, and are applicable to other exemplary embodiments achieved through change, replacement, addition, elimination, or the like where appropriate. Furthermore, constituent elements according to the first to fourth exemplary embodiments can be combined to achieve a new exemplary embodiment. Such other exemplary embodiments will be described below.

The above exemplary embodiments include presenting a message requesting reinput on display14when voice recognition, translation, or reverse translation has a low evaluation value. Controller22can alternatively output voice requesting reinput from speaker unit12, instead of presenting such a message requesting reinput on display14. Speaker unit12exemplifies the notification unit configured differently. The messages depicted inFIG. 4are merely exemplified and may each have different contents.

The above exemplary embodiments merely exemplify the methods of evaluating processing of voice recognition, translation, and reverse translation. These processing results can alternatively be evaluated in accordance with any different method. There may be adopted a method of evaluating whether or not a sentence obtained through each processing is appropriate in its language.

The above exemplary embodiments include calculating the first to fourth evaluation values such that the evaluation values each increase with a more preferred processing result (i.e., with higher evaluation). The present disclosure is not limited by these exemplary embodiments. The first to fourth evaluation values can alternatively be calculated such that the evaluation values each decrease with a more preferred processing result (i.e., with higher evaluation).

The above exemplary embodiments include evaluating “sentence perfection” in accordance with the N-gram model. The present disclosure is not limited by these exemplary embodiments. Such “sentence perfection” can alternatively be evaluated with reference to distributed representation (word vector) (see Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean, “Efficient Estimation of Word Representations in Vector Space”, arXiv: 1301.3781, 2013). It is possible to adopt in combination the neural network method with a single hidden layer, the recurrent neural network method, the convolutional neural network method, the recursive neural network method, or the feedforward neural network method (see Yuta Tsuboi, “Development in deep learning in natural language processing”, Operations Research, 205, 2015). Vectorization of a word or a sentence also exemplifies the method of evaluating sentence closeness.

The second exemplary embodiment includes replacing a word having extremely low appearance probability in previous text. Words can alternatively be compared between the previous text and the current text to select the word having higher appearance probability.

In the above exemplary embodiments, voice recognition server3executes voice recognition, translation server4executes translation, and voice synthesis server5executes voice synthesis. The present disclosure is not limited to this configuration. Translation device1can alternatively execute at least one of voice recognition, translation, and voice synthesis. Furthermore, evaluation server6calculates the evaluation values. Translation device1can alternatively execute calculation of the evaluation values.

The above exemplary embodiments exemplify translation between Japanese and English. The present disclosure is not limited to Japanese and English as languages for translation, and may include any other language (e.g., Chinese, German, French, Spanish, Korean, Thai, Vietnamese, or Indonesian).

Controller22according to the first exemplary embodiment executes processing in step S20if the first evaluation value is less than or equal to the first predetermined value in step S14(seeFIG. 5). Controller22can alternatively execute processing in step S15, regardless of the first evaluation value. If the second evaluation value is less than or equal to the second predetermined value in step S17, controller22may present, on display14, that both voice recognition processing and translation processing are problematic.

Controller22according to the third exemplary embodiment generates the third evaluation value. The present disclosure is not limited to this configuration. The third evaluation value can alternatively be generated by evaluation server6. Generation of the third evaluation value by evaluation server6will exemplarily be described with reference toFIG. 16.FIG. 16depicts translation system100including translation device1and evaluation server6that includes acquisition unit61and evaluator62.FIG. 16does not depict voice recognition server3, translation server4, and voice synthesis server5. Acquisition unit61acquires voice recognition data in Japanese and translation data in English from translation device1. Evaluator62generates an evaluation value for identity between the voice recognition data and the translation data. Similarly to controller22according to the third exemplary embodiment, evaluator62converts the voice recognition data into distributed representation to generate the first distributed representation group. Evaluator62similarly converts the translation data into distributed representation to generate the second distributed representation group. Evaluator62then generates an evaluation value for identity between the first distributed representation group and the second distributed representation group. As described above, evaluation server6may generate the third evaluation value and transmit the third evaluation value to controller22of translation device1via network2. This simplifies the configuration of translation device1functioning as a terminal device.

The exemplary embodiments have been described to exemplify the techniques according to the present disclosure. The accompanying drawings and the detailed description are provided for this purpose.

The constituent elements appearing in the accompanying drawings or the detailed description may accordingly include constituent elements essential for achievement of the object as well as constituent elements inessential for achievement of the object. Such inessential constituent elements should not be recognized as being essential simply because these constituent elements appear in the accompanying drawings or the detailed description.

The above exemplary embodiments are provided to exemplify the techniques according to the present disclosure, and can thus have modification, replacement, addition, removal, and the like in various manners within the scope of the claims or equivalents of the claims.

INDUSTRIAL APPLICABILITY

The present disclosure is applicable to a translation device configured to execute translation according to voice of a speaker.

REFERENCE MARKS IN THE DRAWINGS