Patent Description:
The present disclosure relates to a method and apparatus for reconstructing speaker-specific voice recognition data about voice conversation in a conversation format.

Among techniques for processing natural language inputs, STT (Speech-To-Text) is a voice recognition technique that converts speech into text.

The voice recognition techniques may be classified into two schemes. In a first scheme, all voices to be converted are converted at once. In a second scheme, a voice generated in real time is received on a predetermined time basis, for example, on a basis of a time of less than <NUM> second, and is converted in real time.

In the first scheme, a recognition result is generated after the entire input voice is recognized. In the second scheme, points in time at which the result of voice recognition is generated should be defined.

In the second scheme, there are largely three methods for defining the time points at which recognition results are generated. First, the recognition result may be generated at a time when a special end signal such as recognition/call termination button manipulation is received. Second, the recognition result may be generated at a time when EPD (End Point Detection) occurs, for example, silence lasts for a predetermined time, for example, <NUM> seconds, or more. Third, the recognition result may be generated every predetermined time.

The third method of defining a recognition result generation time is partial in that a time at which the recognition result is generated is a time at which continuous speech is not terminated, that is, in the middle of conversation. Therefore, the third method is mainly used to temporarily obtain a recognition result for a duration from a predetermined point in time to a current time rather than to generate a formal result. Thus, the obtained result is referred to as a partial result.

Unlike the recognition result based on an EPD boundary, the current recognition result of the partial result may include a previously-generated result. For example, in the recognition based on the EPD, results of "A B C," "D E," "F G H" may be generated to recognize "A B C D E F G H". The partial result typically includes previously generated results such as "A," "A B," "A B C," "D," "D E," "F," "F G," and "F G H" as long as EPD does not occur.

Further, the voice recognition technique has recently improved accuracy of voice recognition. However, when recognizing a conversation involving multiple speakers, a voice may not be accurately recognized in the duration for which voices overlap in a situation where two or more persons speak at the same time, and a speaker uttering specific speech may not be accurately identified.

Accordingly, in a commercial system, each input device is used per speaker and voice is recognized per speaker to generate and acquire speaker-specific voice recognition data.

When generating and acquiring voice recognition data for each speaker in a voice conversation, the acquired speaker-specific voice recognition data must be reconstructed in a conversation format. Thus, reconstruction of the speaker-specific voice recognition data in a conversation format is being studied.

Prior art literature includes <CIT>), and <CIT>.

Therefore, the present disclosure has been made in view of the above problems, and it is an object of the present disclosure to provide a voice conversation reconstruction method and apparatus which provide conversation reconstruction as close to the flow of actual conversation as possible in reconstructing speaker-specific voice recognition data about voice conversation in a conversation format.

Objects of the present disclosure are not limited to the above-mentioned objects. Other purposes and advantages in accordance with the present disclosure as not mentioned above will be clearly understood from the following detailed description.

In accordance with a first aspect of the present invention, there is provided a voice conversation reconstruction method performed by a voice conversation reconstruction apparatus, as set forth in independent claim <NUM>.

In accordance with another aspect of the present invention, there is provided a voice conversation reconstruction apparatus, as set forth in independent claim <NUM>.

In accordance with another aspect of the present invention, there is provided a computer-readable recording medium, as set forth in independent claim <NUM>.

In accordance with another aspect of the present invention, there is provided a computer program stored in a computer-readable recording medium, as set forth in independent claim <NUM>.

The same reference numbers in different figures denote the same or similar elements, and as such perform similar functionality. Further, descriptions and details of well-known steps and elements are omitted for simplicity of description. Furthermore, in the following detailed description of the present disclosure, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be understood that the present disclosure may be practiced without these specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the present disclosure.

The terms used in this specification will be briefly described, and then embodiments of the present disclosure will be described in detail.

Although the terms used in this specification are selected, as much as possible, from general terms that are widely used at present while taking into consideration the functions obtained in accordance with at least one embodiment, these terms may be replaced by other terms based on intentions of those skilled in the art, judicial precedent, emergence of new technologies, or the like. Additionally, in a particular case, terms that are arbitrarily selected by the applicant may be used. In this case, meanings of these terms will be disclosed in detail in the corresponding description of the present disclosure. Accordingly, the terms used herein should be defined based on practical meanings thereof and the whole content of this specification, rather than being simply construed based on names of the terms.

It will be further understood that the terms "comprises," "comprising," "includes," and "including" when used in this specification, specify the presence of the stated features, integers, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, operations, elements, components, and/or portions thereof.

Further, as used herein, "unit" means software or hardware such as an FPGA or ASIC. The "unit" performs a specific function. However, the "unit" is not limited to software or hardware. The "unit" may be configured to reside in an addressable storage medium and may be configured to be executed by one or more processors. Thus, in an example, the "unit" may include software, object-oriented software, classes, tasks, processes, functions, attributes, procedures, subroutines, code segments, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The unit or the component may be divided into subunits. Units or components may be combined into a single unit or component.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art may easily implement the present disclosure. In the drawings, portions not related to the description are omitted in order to clearly describe the present disclosure.

<FIG> is a configuration diagram of a voice conversation reconstruction apparatus according to one embodiment.

Referring to <FIG>, a voice conversation reconstruction apparatus <NUM> may include an input unit <NUM> and a processor <NUM>, and may further include an output unit <NUM> and/or a storage <NUM>. The processor <NUM> may include a speaker-specific data processor <NUM>, a block generator <NUM>, a block arranger <NUM>, a block merger <NUM>, and a conversation reconstructor <NUM>.

The input unit <NUM> receives voice conversation. The input unit <NUM> may individually receive voice data about voice conversation per speaker. For example, the input unit <NUM> may include the number of microphones that correspond to the number of speakers in a one-to-one manner.

The processor <NUM> processes voice recognition for the voice conversation as received through the input unit <NUM>. For example, the processor <NUM> may include computing means such as a microprocessor or the like.

The speaker-specific data processor <NUM> of the processor <NUM> acquires speaker-specific voice recognition data about the voice conversation. For example, the speaker-specific data processor <NUM> may include ASR (Automatic Speech Recognition). The ASR may remove noise via preprocessing of the speaker-specific voice data input through the input unit <NUM> and extract a character string therefrom. The speaker-specific data processor <NUM> may apply a plurality of recognition result generation times in obtaining the speaker-specific voice recognition data. For example, the speaker-specific data processor <NUM> may generate a first speaker-specific recognition result about the voice conversation on an EPD (End Point Detection) basis, and generate a second speaker-specific recognition result at each preset time. For example, the second speaker-specific recognition result may be generated after a last EPD at which the first speaker-specific recognition result is generated occurs. In addition, the speaker-specific data processor <NUM> may collect the first speaker-specific recognition result and the second speaker-specific recognition result per speaker without overlap and redundance therebetween to generate the speaker-specific voice recognition data. In another example, the speaker-specific data processor <NUM> may apply a single recognition result generation time point in acquiring the speaker-specific voice recognition data. For example, only one of the first speaker-specific recognition result and the second speaker-specific recognition result may be generated.

The block generator <NUM> of the processor <NUM> divides the speaker-specific voice recognition data acquired by the speaker-specific data processor <NUM> into a plurality of blocks using a boundary between tokens according to a predefined division criterion. For example, the predefined division criterion may be a silent period longer than a predetermined time duration or a morpheme feature related to a previous token.

The block arranger <NUM> of the processor <NUM> may arrange the plurality of blocks divided by the block generator <NUM> in chronological order regardless of the speaker.

The block merger <NUM> of the processor <NUM> may merge blocks related to continuous utterance of the same speaker among the plurality of blocks aligned by the block arranger <NUM>.

The conversation reconstructor <NUM> of the processor <NUM> may reconstruct the plurality of blocks reflecting as merged by the block merger <NUM> in a conversation format based on the chronological order and the speaker.

The output unit <NUM> outputs the processing result from the processor <NUM>. For example, the output unit <NUM> may include an output interface, and may output converted data provided from the processor <NUM> to another electronic device connected to the output interface under the control of the processor <NUM>. Alternatively, the output unit <NUM> may include a network card, and may transmit the converted data provided from the processor <NUM> through a network under the control of the processor <NUM>. Alternatively, the output unit <NUM> may include a display apparatus capable of displaying the processing result from the processor <NUM> on a screen, and may display the voice recognition data about the voice conversation as reconstructed in the conversation format using the conversation reconstructor <NUM> based on the speaker and the chronological order.

The storage <NUM> may store therein an operating system program for the voice conversation reconstruction apparatus <NUM>, and the processing result by the processor <NUM>. For example, the storage <NUM> may include a computer-readable recording medium such as a magnetic medium such as a hard disk, a floppy disk, and magnetic tape, optical media such as a CD-ROM or DVD, a magneto-optical medium such as a floptical disk, and a hardware apparatus specially configured to store and execute program instructions such as a flash memory.

<FIG> is a flowchart for illustration of a voice conversation reconstruction method according to one embodiment. <FIG> is a flowchart illustrating a process of acquiring the voice recognition data per speaker in the voice conversation reconstruction method according to one embodiment. <FIG> is a diagram illustrating a result of the voice conversation reconstruction using the voice conversation reconstruction apparatus according to one embodiment.

Hereinafter, the voice conversation reconstruction method performed by the voice conversation reconstruction apparatus <NUM> according to one embodiment of the present disclosure will be described in detail with reference to <FIG>.

First, the input unit <NUM> individually receives voice data about the voice conversation per speaker, and provides the received speaker-specific voice data to the processor <NUM>.

Then, the speaker-specific data processor <NUM> of the processor <NUM> acquires the speaker-specific voice recognition data about the voice conversation. For example, the ASR included in the speaker-specific data processor <NUM> may remove noise via a preprocessing process of the speaker-specific voice data input through the input unit 110a and may extract the character string therefrom to obtain the speaker-specific voice recognition data composed of the character string S210.

In connection therewith, the speaker-specific data processor <NUM> may apply a plurality of timings at which the recognition result is generated in obtaining the speaker-specific voice recognition data. The speaker-specific data processor <NUM> generates the first speaker-specific recognition result about the voice conversation on the EPD basis. In addition, the speaker-specific data processor <NUM> generates the second speaker-specific recognition result every preset time after the last EPD at which the first speaker-specific recognition result is generated occurs S211. In addition, the speaker-specific data processor <NUM> collects the first speaker-specific recognition result and the second speaker-specific recognition result per speaker without overlap and redundance therebetween, and finally generates the speaker-specific voice recognition data (S212).

The speaker-specific voice recognition data acquired by the speaker-specific data processor <NUM> may be reconstructed into a conversation format later using the conversation reconstructor <NUM>. However, in reconstruction of the data into the conversation format having a text format other than the voice, a situation may occur in which a second speaker interjects during a first speaker's speech. When trying to present this situation in the text format, the apparatus has to determine a point corresponding to the second speaker utterance. For example, the apparatus may divide the entire conversation duration into the data of all speakers based on the silence section, then collect the data of all speakers and arrange the data in chronological order. In this case, when text is additionally recognized around the EPD, a length of the text may be added to the screen at once. Thus, the position in text the user is reading may be disturbed or the construction of the conversation may change. Further, in connection therewith, when a construction unit of the conversation is natural, the context of the conversation is damaged. For example, when the second speaker utters "OK" during the continuous speech from the first speaker, the "OK" may not be expressed in the actual context and may be attached to an end portion of the continuous long word from the first speaker. Further, in connection therewith, in terms of the real time response, the recognition result may not be identified on the screen until EPD occurs even though the speaker is speaking and recognizing the speech. Rather, despite the first speaker speaking first, the word from the second speaker later is short and thus terminates before the speech from the first speaker terminates. Thus, a situation may occur where there is no word from the first speaker on the screen, but only the words from the second speaker are displayed on the screen. In order to cope with these various situations, the voice conversation reconstruction apparatus <NUM> according to one embodiment may execute the block generation process by the block generator <NUM>, the arrangement process by the block arranger <NUM>, and the merging process by the block merger <NUM>. The block generation process and the arrangement process serve to insert the words of another speaker between the words of one speaker to satisfy an original conversation flow. The merging process is intended to prevent a sentence constituting the conversation from being divided into excessively short portions due to generation of blocks as performed for the insertion.

The block generator <NUM> of the processor <NUM> divides the speaker-specific voice recognition data acquired by the speaker-specific data processor <NUM> into a plurality of blocks according to the predefined division criterion, for example, using a boundary between tokens (words/phrases/morphemes) and may provide the plurality of blocks to the block arranger <NUM> of the processor <NUM>. For example, the predefined division criterion may be a silent period longer than or equal to a predetermined time duration or a morpheme feature (for example, between words) related to the previous token. The block generator <NUM> may divide the speaker-specific voice recognition data into a plurality of blocks using the silent section of the predetermined time or longer or the morpheme feature related to the previous token as the division criterion (S220).

Subsequently, the block arranger <NUM> of the processor <NUM> arranges the plurality of blocks generated by the block generator <NUM> in chronological order irrespective of the speaker and provides the arranged blocks to the block merger <NUM> of the processor <NUM>. For example, the block arranger <NUM> may use a start time of each block as the arrangement criterion, or may use a middle time of each block as the arrangement criterion (S230).

Then, the block merger <NUM> of the processor <NUM> may merge blocks from the continuous utterance of the same speaker among the plurality of blocks arranged by the block arranger <NUM>, and may provide the speaker-specific voice recognition data as the results of the block merging to the conversation reconstructor <NUM>. For example, the block merger <NUM> may determine the continuous utterance of the same speaker based on the silent section of a predetermined time duration or smaller between the previous block and the current block or the syntax feature between the previous block and the current block (for example, when the previous block is an end portion of a sentence) (S240).

Next, the conversation reconstructor <NUM> of the processor <NUM> reconstructs the plurality of blocks as the merging result by the block merger <NUM> in the conversation format in the chronological order and based on the speaker, and provides the reconstructed voice recognition data to the output unit <NUM> (S250).

Then, the output unit <NUM> outputs the processing result from the processor <NUM>. For example, the output unit <NUM> may output the converted data provided from the processor <NUM> to another electronic device connected to the output interface under the control of the processor <NUM>. Alternatively, the output unit <NUM> may transmit the converted data provided from the processor <NUM> through the network under the control of the processor <NUM>. Alternatively, the output unit <NUM> may display the processing result by the processor <NUM> on the screen of the display apparatus as shown in <FIG>. As shown in an example of <FIG>, the output unit <NUM> may display the voice recognition data about the voice conversation as reconstructed in a conversation format using the conversation reconstructor <NUM> on the screen in chronological order and based on the speaker. In connection therewith, when updating and outputting the reconstructed voice recognition data, the output unit <NUM> may update and output a screen reflecting the first speaker-specific recognition result generated in step S211. That is, in step S250, the conversation reconstructor <NUM> provides the voice recognition data reflecting the first speaker-specific recognition result to the output unit <NUM> (S260).

Further, each of the steps included in the voice conversation reconstruction method according to the above-described one embodiment may be implemented in a computer-readable recording medium that records therein a computer program including instructions for performing these steps.

Further, each of the steps included in the voice conversation reconstruction method according to the above-described one embodiment may be implemented as a computer program stored in a computer-readable recording medium so as to include instructions for performing these steps.

As described above, according to the embodiment of the present disclosure, in reconstruction of the speaker-specific voice recognition data about the voice conversation in the conversation format, a conversation reconstruction as close as possible to the flow of actual conversation may be realized.

Further, the conversation is reconstructed based on the partial result as the voice recognition result generated every predetermined time during the voice conversation. Thus, the conversation converted in real time may be identified and the real-time voice recognition result may be considered. Thus, an amount of conversation updated once when the voice recognition result is displayed on a screen may be small. Thus, the reconstruction of the conversation may be well arranged, and change in a reading position on the screen is relatively small, thereby realizing high readability and recognition ability.

Combinations of the steps in each flowchart attached to the present disclosure may be performed using computer program instructions. These computer program instructions may be installed on a processor of a general purpose computer, a special purpose computer or other programmable data processing equipment. Thus, the instructions as executed by the processor of the computer or other programmable data processing equipment may create means to perform the functions as described in the steps of the flowchart. These computer program instructions may be stored on a computer-usable or computer-readable recording medium that may be coupled to a computer or other programmable data processing equipment to implement functions in a specific manner. The instructions stored on the computer usable or computer readable recording medium may constitute an article of manufacture containing the instruction means for performing the functions described in the steps of the flowchart. Computer program instructions may also be installed on a computer or other programmable data processing equipment. Thus, a series of operational steps is performed on a computer or other programmable data processing equipment to create a computer-executable process. Thus, instructions to be executed by a computer or other programmable data processing equipment may provide the steps for performing the functions described in the steps of the flowchart.

Further, each step may correspond to a module, a segment, or a portion of a code including one or more executable instructions for executing the specified logical functions. It should also be noted that in some alternative embodiments, the functions mentioned in the steps may occur out of order. For example, two steps shown in succession may in fact be performed substantially simultaneously, or the steps may sometimes be performed in reverse order depending on a corresponding function.

According to one embodiment, in reconstruction of speaker-specific voice recognition data about voice conversation in a conversation format, conversation construction as close to the flow of actual conversation as possible may be provided.

Further, the conversation is reconstructed based on the partial result as the voice recognition result generated every predetermined time during the voice conversation. Thus, the conversation converted in real time may be identified and the real-time voice recognition result may be considered. Thus, an amount of conversation updated once when the voice recognition result is displayed on a screen may be small. Thus, the construction of the conversation may be well arranged, and change in reading position on the screen may be relatively small, thereby realizing high readability and recognizability.

The above description is merely an illustrative description of the technical idea of the present disclosure. A person with ordinary knowledge in the technical field to which the present disclosure belongs will be able to make various modifications and changes.

Claim 1:
A voice conversation reconstruction method performed by a voice conversation reconstruction apparatus (<NUM>), the method comprising:
a0) receiving, by an input unit (<NUM>, 110a), voice conversation of voice conversation per speaker, the input unit including a number of microphones that correspond to the number of speakers in a one-to-one manner such that the input unit (<NUM>) individually receives voice data about the voice conversation per speaker;
a) acquiring (S210) a plurality of speaker-specific voice recognition data corresponding to a plurality of speakers in the voice conversation;
b) dividing (S220), by a block generator (<NUM>), each of the plurality of the speaker-specific voice recognition data into a plurality of blocks using a boundary between tokens depending upon a predefined division criterion;
c) arranging (S230), by a block arranger (<NUM>), the plurality of blocks of each of the plurality of the speaker-specific voice recognition data in chronological order irrespective of a speaker;
d) merging (S240), by a block merger (<NUM>), blocks from continuous utterance of a same speaker among the arranged plurality of blocks; and
e) reconstructing (S250), by a conversation reconstructor (<NUM>), the plurality of blocks subjected to the merging in a conversation format in chronological order and based on a speaker,
f) wherein acquiring (S210) the speaker-specific voice recognition data includes:
f1) acquiring (S211) a first speaker-specific recognition result generated on an End Point Detection, EPD, basis from the voice conversation and a second speaker-specific recognition result generated every preset time from the voice conversation; and
f2) collecting (S121) the first speaker-specific recognition result and the second speaker-specific recognition result without overlap and redundance therebetween to generate the speaker-specific voice recognition data,
g) wherein the second speaker-specific recognition result is generated after a last EPD at which the first speaker-specific recognition result is generated occurs,
h) wherein the chronological order is determined by using a start time of each of the plurality of blocks of each of the plurality of the speaker-specific voice recognition data, and
i) wherein in step e), the conversation reconstructor (<NUM>) provides (S260) the voice recognition data reflecting the first speaker-specific recognition result to an output unit (<NUM>).