Voice communication method and apparatus

Abstract: The invention relates to voice communication method and apparatus and it is an object that when the acceptable vocabulary or grammar is dynamically changed in accordance with a communication state, a troublesomeness for allowing the user to again utter a speech which does not correspond to the predicted vocabulary and which cannot be acceptable is eliminated. For the above purpose, the speech which cannot be perceived is handled as an unknown word, a question to induce an answer such that the unknown word becomes included the acceptable vocabulary is performed, and the unknown word portion is reevaluated at a time point when the unknown word becomes included the acceptable vocabulary due to the induced answer. Thus, the user does not need to again utter the same answer.

Description: BACKGROUND OF THE INVENTION Field of the Invention The invention relates to a method and apparatus for recognising input speech and for obtaining the speaking intention of the speaker. The invention also relates to method and apparatus for recognising a speech in consideration of the communication situation with a speaker. Related Background Art i) Hitherto, in the speech perception in the voice communication, a high efficiency of processes is realized by limiting the vocabulary, sentences, or the like which can be received. On the other hand, there is a technique such that the user predicts words to be uttered next from a flow of communication, thereby effectively limiting the vocabulary, sentence, or the like. ii) There is also a technique which models vocabulary, sentences, and the like which cannot be accepted and used them together, and detecting cases where the user utters the modelled vocabulary, sentences, or the like. iii) Hitherto, there is a technique such that for a word which cannot be recognised, the user is prompted to again utter the same word and the speech which was again input is changed to a different recognition method. According to the conventional technique (i), even if the speech which will be uttered next by the user is effectively predicted, a possibility such that the user will utter another vocabulary, sentence, or the like which cannot be accepted still remains. Now, even if the models of the vocabulary, sentence, and the like which cannot be accepted are used by the conventional technique (ii), when the user utters the vocabulary, sentence, or the like which cannot be accepted, a fact that "the user uttered vocabulary, sentence, or the like which cannot be accepted" can be merely detected and which word the user uttered cannot be recognised. Therefore, after the acceptable vocabulary, sentence, or the like was modified, the user needs to utter again the content which was previously uttered. There is a problem such that such an operation is difficult for the user and any feeling of ease of use is lost. According to the conventional technique (iii), the user must repetitively utter the same word many times until the system can accurately recognise it, so that ease of use is lost. In accordance with a first aspect of the present invention there is provided a voice recognition method as set out in claim 1. In accordance with a further aspect of the invention there is provided voice recognition apparatus as set out in claims 6 and 11. SUMMARY OF THE INVENTION The invention intends to provide voice communication method and apparatus in which the acceptable vocabulary is dynamically changed in accordance with a communication situation, an unknown word, which cannot be recognised by the current acceptable vocabulary, is detected, and the unknown word is reevaluated by the acceptable vocabulary changed by the information obtained after a speech including the unknown word which had been detected was input. Due to this, even when vocabulary, sentence, or the like which cannot be accepted is input, the processes continue, the past speech is reevaluated, the problem to the user such that "after the acceptable vocabulary, sentence, or the like was changed, the user must again utter the contents which had been once uttered" is solved, and the use feeling of the user can be improved. BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 is a flowchart showing a whole process of the embodiment; Fig. 2 is a block diagram showing a construction of an apparatus of the embodiment; Fig. 3 is a diagram showing an example of acceptable vocabulary, sentences, and the like; Fig. 4 is a diagram showing an example of paths obtained as a result in speech perceiving step S3; Fig. 5 is a diagram showing the correspondence between the speech and the perception result in the embodiment 1; Fig. 6 is a diagram showing an example of paths obtained as a result in speech perceiving step S3; Fig. 7 is a diagram showing an example of information obtained as a result of the data base search in communication processing step S5; Fig. 8 is a diagram showing an example of acceptable vocabulary, sentence, and the like which were changed as a result in acceptable vocabulary or sentence changing step S6; Fig. 9 is a diagram showing an example of information obtained as a result of the data base search in communication processing step S5; Fig. 10 is a diagram showing an example of acceptable vocabulary, sentences, and the like which were changed as a result in acceptable vocabulary or sentence changing step S6; Fig. 11 is a diagram showing an outline of the process in the embodiment 1; Fig. 12 is a diagram showing an outline of the process in the embodiment 2; Fig. 13 is a diagram showing an example of a communication processing flow using a category to which an unknown word belongs in communication processing step S5; Fig. 14 is a diagram showing an example of a table which is used to decide an action to follow in step S10 to determine how to process an unknown word; Fig. 15 is a diagram showing an example of communication which is used for explanation of the embodiment 1; and Fig. 16 is a diagram showing an example of vocabulary to know a category of an unknown word. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS [Embodiment 1] An embodiment of the invention will now be described hereinbelow with reference to the drawings. Fig. 1 is a flowchart showing a processing flow of an embodiment of the invention. Fig. 2 is a diagram showing a hardware construction in an embodiment of the invention. In Fig. 2, reference numeral H1 denotes an output device such as display device like a CRT display or the like or speech synthesizing device; H2 a speech input device such as a microphone or the like; H3 an I/O device to modify a speech which is input from the speech input device H2 so that it can be processed by a computer; and H4 a central processing unit (CPU) for executing processes such as numerical value calculation, control, and the like. The CPU H4 controls each process in accordance with a control program stored in a memory device H5. H5 denotes the memory device such as external memory device like a disk device or the like or internal memory like an RAM, ROM, or the like. The memory device H5 stores a dictionary of speech which is used for speech perception. A control program of processes shown in flowcharts, which will be explained hereinbelow, has been stored in the memory device H5. Each of the above devices H1 to H5 is connected through a bus. In consideration of the above hardware construction, an embodiment of the present invention will now be described with reference to the flowchart of Fig. 1. When a signal to instruct the start of processes is input to the CPU H4, first, in step S1, a vocabulary, sentence, or the like which is first uttered by the user is predicted and the predicted vocabulary, sentence, or the like is stored into the memory device H5. In step S2, the user inputs a speech as a recognition target by using the speech input device H2. In step S3, speech recognition is executed to the speech which was input in step S2 by using the vocabulary, sentence, or the like which was decided in step S1. In step S4, a check is made to see whether or not an unknown word in which it is determined such that the recognition result is defective and that the word or the like which is derived as a recognition result cannot be output is included in the speech which was input in step S2. Further, in the case where an unknown word is included, the position of the unknown word portion in the speech data which was modified by the I/O device H3 is detected and the position information or the speech data which was judged as an unknown word itself is stored into the memory device H5. In step S5, the results in steps S3 and S4 are analyzed, the speaking intention of the user is extracted, and a process such as a data base search or the like which the user intends to execute is performed in order to communicate for the speaking intention. In step S5, a process in the case where there is an unknown word when performing the process to extract the speaking intention in step S5 is shown in a flowchart of Fig. 13 and will be explained in detail hereinbelow. In step S10, in the case where the unknown word is detected in step S4, a method for eliminating the unknown word and correctly perceiving the speech is decided or planned by using the result in step S5. In step S6, the vocabulary, sentences, and the like which are recognised and have been stored in the memory device H5 in step S1 are changed by using the result in step S5. In step S7, whether the unknown word portion detected in step S4 should be reevaluated or not is decided. In order to judge the necessity about the reevaluation, information indicating whether the unknown word has been detected in step S4 or not, whether the acceptable vocabulary, sentence, or the like has been changed in step S6 or not, or the like is read out from the memory device H5 and is used. In the case where there is no unknown word as a result in step S7 or where there is an unknown word and it is determined that there is no need to change the acceptable vocabulary or sentence in step S6 and there is no need to reevaluate the unknown word portion, the processing routine advances to step S8. In the case where there is an unknown word and it is decided that the acceptable vocabulary or sentence has been changed in step S6 and it is necessary to reevaluate the unknown word portion, the processing routine advances to step S9. In step S8, the reply contents to the user are calculated by using the results in steps S5 and S10 and are shown to the user by displaying on the display screen or synthesizing the speech and generating as a sound or the like. After step S8, the processing routine is returned to step S2. In the case where it is determined in step S7 that it is necessary to reevaluate the unknown word portion, the processing routine advances to step S9. In step S9, the unknown word portion detected in step S4 is reevaluated (namely, the speech recognition of the unknown word portion is again executed) detected in step S4 by using the acceptable vocabulary or sentence which was changed in step S6. After completion of step S9, the processing routine is returned to step S4. As mentioned above, since whether or not an unknown word is included in the recognition result is detected in step S4, a check is made to see if any other unknown word still remains or not in the result in step S9. In step S5, since a proper process is executed by using the result of the speech recognition in the case where the unknown word has already been eliminated in step S9, a process is executed by using new information of the portion in which the unknown word existed so far. The above processing procedure will now be described by using a communication example shown in Fig. 15. Fig. 15 shows a communication example such that since the user (usr) first utters a speech including the names of cities, towns, and villages which cannot be accepted for the first time because they are not in the acceptable vocabulary, namely, when the user first utters " (I WOULD LIKE TO KNOW ABOUT SPA SITES IN HAKONE TOWN.)", " (HAKONE TOWN)" is perceived as an unknown word of the place name. The system (sys) replies so that the place name can be determined in response to it, thereby generating an answer which the user demands. Fig. 11 is a table simply showing a state of data at each stage to process the communication of Fig. 15. The first line in Fig. 11 shows the result until "sys1" is uttered after "usr1" was input. The second line shows the result in which "usr2" was input. The third line shows a state when "sys2" is uttered. The above data is stored and updated in the RAM of the memory device H5 every kind. The detailed processes of the communication of Fig. 15 will now be described in accordance with the flowchart of Fig. 1. First, in step S1, a vocabulary or sentence which will be first uttered by the user is predicted and the acceptable vocabulary or sentence is determined. For example, the first acceptable vocabulary or sentence is as shown in Fig. 3. The meaning of the diagram of the sentence of Fig. 3 is that a sentence which traces paths from "start" to "end" is accepted. The meaning of the diagram of the vocabulary shows a vocabulary which can become "$hello" or the like in the sentence. Namely, (GOOD MORNING)", " (I WOULD LIKE TO KNOW HOW TO GET AN ART MUSEUM IN TOKYO.)", " (THIS IS THE PHONE #.)", and the like are sentences which can be accepted. On the other hand, (UNKNOWN) included in "$place" or the like in the diagram of the vocabulary in Fig. 3 is a symbol which indicates an unknown word and is provided to detect an unknown word in step S4. That is, it means "unknown word can enter here.". In the next step S2, the user inputs a speech. It is now assumed that the user input, for example, " (I WOULD LIKE TO KNOW ABOUT SPA SITES IN HAKONE TOWN.)" (usr1). In step S3, as a path (including vocabulary) on which the speech which was input in step S2 on the sentence decided in step S1, the optimum path which can obtain the speech that is acoustically nearest is calculated. The optimum path obtained is set to arecognition result. In the above example, since the user utters " (I WOULD LIKE TO KNOW ABOUT SPA SITES IN HAKONE TOWN.)", a path shown by a solid line in Fig. 4 is obtained as a recognition result. " (I WOULD LIKE TO KNOW ABOUT SPA SITES IN (UNKNOWN).)" is stored as a recognition result of (usr1) as shown in Fig. 11. In step S4, a check is made to see if an unknown word is included in the speech which was input in step S2 or not on the basis of the result in step S3. As mentioned above, since "(unknown word)" is included in the recognition result which was obtained and stored as a result in step S3, it can be judged that the unknown word is included in the speech which was input in step S2. The position (gray portion in Fig. 5) of the unknown word as shown in Fig. 5 is detected from the boundary of the vocabulary derived as a result in step S3. In step S5, the speaking intention of the user is first extracted from the results in steps S3 and S4. The result in step S3 is " (I WOULD LIKE TO KNOW ABOUT SPA SITES IN (UNKNOWN).)". From (I WOULD LIKE TO KNOW ABOUT SPA SITES.)", the speaking intention is understood to be that "the search target of the user is spa sites.". Also, from " (BE IN)" of " (BE IN (UNKNOWN))" is an annexed word indicative of the place, it is obtained that "although the user designates the place, it is unknown word.". The information obtained is stored as "information obtained" of "usr1" as shown in Fig. 11. As mentioned above, since the category of the unknown word is known, the speaking intention of the user is more finely extracted by using the category of the unknown word. The process for using the category of the unknown word is as shown in, for instance, a flowchart of Fig. 13. In step S101, the category of the unknown word is judged from the word which continues to the unknown word. That is, as already described above, since " (BE IN)" of " [BE IN (UNKNOWN)]" is the annexed word indicative of the place, it is judged that the unknown word is a word indicative of the place. In step S102, the information is stored as "information obtained" of "usr1". The processing routine advances to step S6. The annexed word and condition for judgment in step S101 are previously stored in the memory device H5 and are sequentially collated, thereby judging. The information of "place = (unknown)" indicates that "the place as a target to be searched by the user is handled as (unknown).", namely, "the subsequent processes are executed by using that the user designated the place and the designated place is unknown.". Subsequently, the process such as data base search or the like which the user intends to perform is executed from the speaking intention. The contents which are calculated from the speaking intention are "do nothing". This is because some condition must be added in order to search the spa sites by the data base and it is intended to promote the user so as to add the conditions. In step S10, since the unknown word has been detected in step S4, an action to eliminate the unknown word is decided. Since it is known that the unknown word indicates the place name by the result in step S5 stored in the memory device H5, an action to eliminate the unknown word of the place name is determined. For example, in case of the place name, a situation such that it was detected as an unknown word denotes that it is possible to predict that the place name which was uttered by the user is other than the names of prefectures (the names of prefectures are always stored in the acceptable vocabulary in the embodiment). Therefore, it is considered that when the user designates the name of prefecture, the unknown word can be eliminated. Therefore, the operation such that "ask the name of prefecture to the user." is used as an action to follow. In order to decide the action to follow, for example, data for reference as shown in a table of Fig. 14 is stored into the memory device H5 and is referred in order to decide the action to follow from the unknown word category and additional condition in step S10. Step S6 follows. However, since the data base search is not performed in step S5, the acceptable vocabulary or sentence is not changed. In step S7, whether the unknown word portion is reevaluated or not is decided. As mentioned above, as a result in step S4, the unknown word exists in the speech which was input in step S2. However, since the acceptable vocabulary or sentence is not changed in step S6, it is judged that there is no need to reevaluate the unknown word portion. Thus, the processing routine advances to step S8. In step S8, a proper reply is formed on the basis of the results in steps S5 and S10. The result in step S5 has the contents that "it is necessary to add conditions for data base search.". The action to follow which was obtained as a result in step S10 is that "ask the name of prefecture to the user.". For example, therefore, a reply sentence such that " (PLEASE DESIGNATE THE NAME OF PREFECTURE.)" is returned. After such a reply sentence was proposed, the processing routine is returned to step S2. In step S2 again, an example of the user input at the second time assumes " (IT IS KANAGAWA PREFECTURE.)" (usr2). The result in step S3 for such an input is as shown in Fig. 6 and is stored as a recognition result of "usr2" as shown in Fig. 11. Step S4 now follows. As mentioned above, since "unknown" is not included in the result in step S3, the unknown word is not detected. In step S5, the speaking intention is first extracted from the result in step S3 and the result in step S5 until the preceding time. As mentioned above, since the result in step S3 is such that " (IT IS KANAGAWA PREFECTURE.)", information that "the place designated by the user is Kanagawa prefecture" is obtained and is additionally stored into the information which is obtained as shown in Fig. 11. As a result in step S5 at the preceding time, it is obtained that "the search target of the user is spa sites.". Therefore, "the spa sites located at places in Kananagawa prefecture." is searched from the data base. Fig. 7 shows an example of the result of the data base search. In the example, the places and names of spa sites in Kanagawa prefecture are searched. In step S10, since no unknown word is detected in step S4, nothing is performed. In step S6, the acceptable vocabulary or sentence is changed on the basis of the result in step S5, namely, Fig. 7 in this case. For example, the names of cities, towns, and villages in which spa sites exist and the names of spa sites are shown in Fig. 7. A possibilitv such that those vocabularies are used in the speech recognition after that is high. On the other hand, although the names of spa sites are not clearly shown in Fig. 7, since the number of spa site names is large, it is not adapted to the action for suppressing the number of acceptable words. Therefore, as an example of the operation in step S6, the operation such that "the names of cities, towns, and villages in which spa sites exist are added to predicted vocabulary" is executed. With respect to the sentence, since the name of city, town, or village can be used as it is as "$place", it is not changed. Consequently, a new acceptable vocabulary or sentence is as shown in Fig. 8. The processing routine advances to step S7. First, since the unknown word exists in the speech " (I WOULD LIKE TO KNOW ABOUT SPA SITES IN HAKONE TOWN.)" which was input in step S2 at the first time, there is a target for reevaluation. On the other hand, since the acceptable vocabulary has been changed in step S6 just before, it is considered that the unknown word can be eliminated by reevaluating the unknown word portion by using the new acceptable vocabulary or sentence. Therefore, the processing routine advances to step S9. In step S9, the unknown words which have been uttered so far are eliminated by the reevaluation by using the new acceptable vocabulary or sentence. In step S4 at the first time, the position of the unknown word has already been obtained (in Fig. 5). It has been known that the gray portion in Fig. 5 relates to the place. Therefore, the speech information at the position of the unknown word is read out from the memory device H5 and is matched with the vocabulary indicative of the place name which was newly added in step S6. The speech that is acoustically nearest to the gray portion in Fig. 5 is selected and is set to the result of the reevaluation of the unknown word. In the example, " (HAKONE TOWN)" is obtained as a result in step S9. it will be understood that " (HAKONE TOWN)" obtained here is acoustically near to the gray portion in Fig. 5 than the model "unknown" indicative of the unknown word. In step S4, since there is no model indicative of the unknown word in the new speech recognition result" (I WOULD LIKE TO KNOW ABOUT SPA SITES IN HAKONE TOWN.)" obtained in step S9, a result such that "the unknown word is not detected" is obtained. In step S5, the speaking intention that "the target place name of the user is Hakone town." is obtained from the new speech recognition result " (I WOULD LIKE TO KNOW ABOUT SPA SITES IN HAKONE TOWN .)" obtained in step S9. As a result in step S5 until the preceding time, information that "the search target of the user is spa site" has been obtained. Therefore, "spa sites located in Hakone town" are searched from the data base. An example of the result of the data base search is shown in Fig. 9. In the example, the places and names of the spa sites in HAKONE town are searched. In the next step S10, nothing is performed because no unknown word is detected in step S4. The acceptable vocabulary or sentence is subsequently changed on the basis of the result (Fig. 9) in step S5. Different from the case of Fig. 7, since the number of spa site names is small, the spa site names are added to the acceptable vocabulary. The sentence is also changed in association with the addition of the spa site names (proper nouns). As a result of the above processes, the new acceptable vocabulary or sentence is as shown in Fig. 10. In step S7, since no unknown word is left in the speech which was input in the past, the processing routine advances to step S8. As a result in step S5, the information (Fig. 9) which satisfies the contents such as " (I WOULD LIKE TO KNOW ABOUT SPA SITES LOCATED IN HAKONE TOWN.)" which the user intended is obtained. Therefore, in step S8, such information is proposed. For example, a reply such as " (THERE ARE TOTAL 13 SPA SITES SUCH AS ASINOYU SPA, SOKOKURA SPA, ETC.)" (sys2) is proposed to the user. After that, the processing routine again advances to step S2. [Embodiment 2] In the embodiment 1, a format such as "unknown" has been used as a model indicative of the unknown word. However, particularly, in case of a proper noun or the like, the unknown word often has a character train indicative of the category of such a word in the end portion. Therefore, a model such as "(unknown) + (word indicative of the category)" can be also used as a model showing the unknown word. For example, by possessing the model such as "(unknown) + (word indicative of the category)" as shown in Fig. 6 as a vocabulary, the category of the unknown word can be known together with the unknown word. Namely, by expressing the unknown spa site name by "(unknown) spa", it will be understood that the category of the unknown word is a spa site. The process to judge the category of the unknown word is executed in step S5 and is shown in detail in the flowchart of Fig. 13. In step S101, in the case where it is judged that the user designated the name in the portion of the unknown word from the characteristics of the word which continues to the unknown word, step S103 follows. When the word which continues to the unknown word is " (ART MUSEUM)", "category for search = art museum, name = (unknown)" is stored as "information obtained" of "usr1". In the case where it is judged that the word which continues to the unknown word is not "art museum" in step S103 and is judged that it is spa in step S105, "category for search = spa, name = (unknown)" is stored as "information obtained" of "usr1". In the case where it is judged in step S105 that the word which continues to the unknown word is not "spa" and is judged in step S107 that it is the word indicative of a temple or shrine, "category for search = temple & shrine, name = unknown" is stored as "information obtained" of "usr1". By repeating the processes for judging "(unknown) + (word indicative of the category)" in steps S103, S105, and S107, if the answers are NO in all of those judging steps, as information of the unknown word, "category for search = (unknown), name = (unknown)" is stored as "information obtained" of "usr1" in step S109. The category of the unknown word is known by the processes shown in Fig. 13 and the user doesn't need to again utter about the category. Processes shown in Fig. 12 will now be described. Fig. 12 relates to the processes in case of the following communication and in the case where there is a vocabulary such as "(unknown) + art museum". usr1: (I WOULD LIKE TO KNOW HOW TO GET TO CHOKOKUNOMORI ART MUSEUM.) sys1: (PLEASE DESIGNATE THE PREFECTURE.) usr2: (KANAGAWA.) sys2: (YOU CAN GET TO CHOKOKUNOMORI ART MUSEUM IN 2 MIN. ON FOOT FROM CHOKOKUNOMORI STATION.) As another example of "(unknown) + (word indicative of the category)", it is possible to mention "(unknown) country club", "(unknown) prefecture" "(unknown) hotel", and the like. [Embodiment 3] In step S9 in the embodiment 1, only the portion of the unknown word (gray portion in Fig. 5) has been reevaluated. However, the whole uttered speech including the unknown word can be also reevaluated by using the new acceptable vocabulary or grammar obtained in step S6. According to this method, an error of a vocabulary boundary due to the mixture of an unknown word can be corrected.

Claims: A voice recognition method in which an acceptable speech vocabulary is dynamically changed in accordance with a communication situation, comprising the steps of : detecting from an input speech an unknown word which cannot be recognised by the acceptable speech vocabulary; and reevaluating the unknown word portion by the acceptable speech vocabulary as changed by information obtained after the speech which included the detected unknown word had been input. A method according to claim 1, wherein said reevaluation of the unknown word portion is executed with respect to an input speech information which had been input before the just previous speech information had been input. A method according to claim 1, wherein said information is a category information of a category to which the unknown word belongs, and wherein this information is extracted from the speech information including the unknown word. A method according to claim 3, wherein the communication situation is judged by determining the category to which the unknown word belongs. A method according to claim 4, wherein a reply sentence to eliminate the unknown word is determined by the judged communication situation. A speech communication apparatus comprising: changing means for dynamically changing an acceptable speech vocabulary in accordance with a communication situation; and unknown word detecting means for detecting an unknown word which cannot be perceived by the acceptable speech vocabulary from an input speech; reevaluating means for reevaluating the unknown word portion by the acceptable speech vocabulary changed by information obtained after the speech including the detected unknown word had been input. Apparatus according to claim 6, wherein said reevaluating means executes a reevaluation of the unknown word portion with respect to an input speech information which had been input before the just previous speech information had been input. Apparatus according to claim 6, wherein said information is a category information of a category to which the unknown word belongs, and wherein this information is extracted by a category extracting means from the speech information including the unknown word. An Apparatus according to claim 8, further having judging means adapted to judge the communication situation by determining the category to which the unknown word belongs. Apparatus according to claim 9, further having reply sentence deciding means for deciding a reply sentence to eliminate the unknown word by judging the communication situation. Speech recognition apparatus comprising means for detecting speech input unknown to a data base in the apparatus, means for requesting verbal clarification, and means for altering the data base of the apparatus in response to the clarification.

Inventors: KOMORI YASUHIRO, OHORA YASUNORI, YAMADA MASAYUKI, KOMORI, YASUHIRO, OHORA, YASUNORI, YAMADA, MASAYUKI, KOMORI, YASUHIRO, C/O CANON KABUSHIKI KAISHA, OHORA, YASUNORI, C/O CANON KABUSHIKI KAISHA, YAMADA, MASAYUKI, C/O CANON KABUSHIKI KAISHA
Applicants: CANON KK, CANON KABUSHIKI KAISHA
Application year: 1994
Main classification: G10L5
IPC codes: G10L15