MOTION SENSOR-BASED PORTABLE AUTOMATIC INTERPRETATION APPARATUS AND CONTROL METHOD THEREOF

Disclosed herein is a motion sensor-based portable automatic interpretation apparatus and control method thereof, which can precisely detect the start time and the end time of utterance of a user in a portable automatic interpretation system, thus improving the quality of the automatic interpretation system. The motion sensor-based portable automatic interpretation apparatus includes a motion sensing unit for sensing a motion of the portable automatic interpretation apparatus. An utterance start time detection unit detects an utterance start time based on an output signal of the motion sensing unit. An utterance end time detection unit detects an utterance end time based on an output signal of the motion sensing unit after the utterance start time has been detected.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

The present invention is intended to detect a behavioral pattern frequently expressed by a user, who utilizes an automatic interpretation system in a portable device, in the form of motion, and to allow the automatic interpretation system to automatically detect the start time and end time of speech based on such motion when the user normally speaks and when the user talks with a person who uses another language.

Hereinafter, a motion sensor-based portable automatic interpretation apparatus and control method thereof according to embodiments of the present invention will be described in detail with reference to the accompanying drawings. Prior to the detailed description of the present invention, it should be noted that the terms or words used in the present specification and the accompanying claims should not be limitedly interpreted as having their common meanings or those found in dictionaries. Therefore, the embodiments described in the present specification and constructions shown in the drawings are only the most preferable embodiments of the present invention, and are not representative of the entire technical spirit of the present invention. Accordingly, it should be understood that various equivalents and modifications capable of replacing the embodiments and constructions of the present invention might be present at the time at which the present invention was filed.

FIG. 1is a block diagram showing a motion sensor-based portable automatic interpretation apparatus according to an embodiment of the present invention, andFIGS. 2 and 3are diagrams showing the references of X, Y, and Z axes used in the embodiment of the present invention.

A motion sensor-based portable automatic interpretation apparatus according to an embodiment of the present invention includes a motion sensing unit10, an utterance start time detection unit12, an utterance end time detection unit14, a language pair change detection unit16, a microphone18, a speech recognition unit20, a translation unit22, a Text to Speech (TTS) unit24, and a speaker26.

The motion sensing unit10senses the motion of the portable automatic interpretation apparatus and outputs sensor result values of the X, Y and Z axes (for example, gravity acceleration values) occurring as the user holds and moves the portable automatic interpretation apparatus. Preferably the motion sensing unit10includes one or more of an acceleration sensor and a gyro sensor (gyroscope). For example, a 3-axis acceleration sensor may be used, and a 2-axis acceleration sensor and a 1-axis gyro sensor may be used together. Of course, in addition to the acceleration sensor and the gyro sensor, any type of sensor may be employed in the present invention as long as it is capable of sensing the motion of the portable automatic interpretation apparatus. Which motion was conducted by the portable automatic interpretation apparatus can be detected based on the sensor result values of the X, Y, and Z axes output from these sensors.

Meanwhile, the portable automatic interpretation apparatus may be a smart phone, such as an Android phone or an iPhone. Such a smart phone contains therein an automatic interpretation system. Consequently, the smart phone containing the automatic interpretation system may be referred to as the portable automatic interpretation apparatus, or the automatic interpretation system in the smart phone may also be referred to as the portable automatic interpretation apparatus.

The references of the X, Y, and Z axes in the embodiment of the present invention will be described below. InFIGS. 2 and 3, when a portable automatic interpretation apparatus50is viewed in a front direction (that is, viewed from a surface on which a touch screen28is present), a lateral direction corresponds to an X axis and a vertical direction corresponds to a Y axis. When the portable automatic interpretation apparatus50is viewed from a side surface, an axis in a direction extending from the rear surface of the portable automatic interpretation apparatus50to the front surface thereof corresponds to a Z axis. In other words, it may be considered that the X axis corresponds to the width direction of the portable automatic interpretation apparatus50, the Y axis corresponds to the longitudinal direction of the portable automatic interpretation apparatus50, and the Z axis corresponds to the height direction of the portable automatic interpretation apparatus50.

InFIG. 1, the utterance start time detection unit12detects an utterance start time based on the output signal of the motion sensing unit10. If the utterance start time is detected by the utterance start time detection unit12, the speech recognition unit20receives microphone input for speech recognition. The utterance start time detection unit12detects time at which the output signal of the motion sensing unit10, corresponding to motion in which the portable automatic interpretation apparatus enters a horizontal state with respect to the Y axis, is input, as an utterance start time. In greater detail, the utterance start time detection unit12is configured to, when the output signal of the motion sensing unit10is a signal corresponding to time at which the Y axis of the portable automatic interpretation apparatus has moved in a horizontal direction and the Z axis has moved in a vertical direction and thereafter the movement deviation of the portable automatic interpretation apparatus rapidly decreases, receive the signal as a command indicating the start of an utterance.

The utterance start time detection unit12may previously store a table in which the sensor result values of the X, Y, and Z axes are combined and different contents for the combined sensor result values of respective X, Y, and Z axes are recorded. Accordingly, the utterance start time detection unit12may decode the meaning of the output signal of the motion sensing unit10based on the information stored in the table and detect the utterance start time.

The utterance end time detection unit14detects an utterance end time based on the output signal of the motion sensing unit10in a state in which the detection of the utterance start time is performed by the utterance start time detection unit12and the speech of the user is being transferred to the speech recognition unit20via the microphone18. Preferably, in the state in which the utterance start time has been detected by the utterance start time detection unit12and the speech of the user is transferred to the speech recognition unit20through the microphone18, the utterance end time detection unit14detects time, at which the output signal of the motion sensing unit10, corresponding to time at which the portable automatic interpretation apparatus moves from the horizontal state to a vertical direction with respect to the Y axis and thereafter has less motion deviation, is input, as an utterance end time.

The utterance end time detection unit14may previously store a table in which the sensor result values of the X, Y, and Z axes are combined and different contents for the combined sensor result values of respective X, Y, and Z axes are recorded. Accordingly, the utterance end time detection unit14may decode the meaning of the output signal of the motion sensing unit10based on the information stored in the table and detect the utterance end time.

The language pair change detection unit16is configured to, when there is an output signal corresponding to time at which inversion between the sensor result value of the X axis and the sensor result value of the Y axis has greatly occurred in the output signal of the motion sensing unit10and then the motion deviation of the portable automatic interpretation apparatus is decreased, recognize the corresponding output signal as a language pair change command. In this way, if the language pair change detection unit16recognizes the language pair change command and sends a signal corresponding to the command to the speech recognition unit20, the speech recognition unit20exchanges a source language and a target language for each other, and performs speech recognition. The change of a pair of a source language and a target language will be described by way of example. When a first user sets Korean as a source language and English as a target language, the first user utters Korean and transfers Korean speech to the portable automatic interpretation apparatus, and the portable automatic interpretation apparatus outputs the target language, that is, English speech, through the speaker26. In the case where the native language of a second user is English and the target language thereof is Korean, if a language pair change command is recognized, the portable automatic interpretation apparatus changes the source language to English and the target language to Korean. Accordingly, when the second user speaks in English and speech data is transferred to the portable automatic interpretation apparatus, the portable automatic interpretation apparatus outputs the corresponding speech data in Korean.

The language pair change detection unit16may previously store a table in which the sensor result values of the X, Y, and Z axes are combined and different contents for the combined sensor result values of respective X, Y, and Z axes are recorded. Accordingly, the language pair change detection unit16may decode the meaning of the output signal of the motion sensing unit10based on the information stored in the table, and detect (recognize) the language pair change command.

The microphone18is installed in the portable automatic interpretation apparatus and collects speech utterances made by the user.

The speech recognition unit20converts speech data input through the microphone18into text based on the utterance start time detection signal output from the utterance start time detection unit12and the utterance end time detection signal output from the utterance end time detection unit14. That is, when utterance is initiated, the speech recognition unit20receives the speech utterance of the user through the microphone18and stores the speech utterance. Then, when the utterance is terminated, the speech recognition unit20converts stored speech data into text. The speech recognition unit20may be regarded as performing speech recognition via a speech recognition program. It may be considered that details of speech recognition can be sufficiently understood by those skilled in the art based on well-known technology.

The translation unit22outputs results obtained by translating text (that is, text in a source language) output from the speech recognition unit20into a target language (that is, text). It may be considered that details of the translation process of the translation unit22can be sufficiently understood by those skilled in the art based on well-known technology.

The Text-to-Speech (TTS) unit24converts the text output from the translation unit22into speech and outputs the speech to the outside of the interpretation apparatus through the speaker26.

Hereinafter, a method of controlling motion sensor-based automatic interpretation according to an embodiment of the present invention will be described in detail with reference to the flowchart ofFIG. 4.

At step S10, the motion sensing unit10senses the motion of the portable automatic interpretation apparatus.

At step S12, the utterance start time detection unit12detects an utterance start time based on the sensor result values of the X, Y, and Z axes output from the motion sensing unit10.

In the past, since the utterance start time cannot be automatically detected, a command for indicating the start of utterance is transferred to the portable automatic interpretation apparatus by touching a specitic button on the screen of the apparatus, but the present invention focuses on the action of moving the microphone18close to the face of a user, in particular, the mouth of the user, so as to transfer accurate speech to the apparatus.

When a value indicated by the motion sensing unit10represents time at which motion in which the front of the apparatus (that is, the surface on which a touch screen is present) faces a ground surface, or the reverse motion thereof is detected, the utterance start time detection unit12of the present invention recognizes the time as an utterance start time.FIGS. 5 and 6illustrate the initial status of the portable automatic interpretation apparatus50, that is, the incline and movement status of the portable automatic interpretation apparatus50which waits for an utterance start command. This status may be a state in which the user holds the rear surface of the portable automatic interpretation apparatus50with his or her hand, or a state in which the portable automatic interpretation apparatus50is supported by a specific object. In such initial status, in order for the user to initiate utterance, the user leans and moves the portable automatic interpretation apparatus50in the direction of arrow A to move a portion of the microphone18close to the face (in detail, mouth) of the user. If the microphone18is moved close to the mouth of the user in this way, the status illustrated inFIG. 7is obtained. That is, depending on the location of the microphone18, the front surface (the surface on which the touch screen28is present) and the rear surface may be reversed. However, the utterance start time detection unit12receives motion, which makes the portable automatic interpretation apparatus50be in a state close to a vertical direction with respect to the Z axis and a horizontal direction with respect to the Y axis, as a command for indicating the start of utterance. In other words, the utterance start time detection unit12is configured to, if time at which the Z axis of the portable automatic interpretation apparatus50has moved in the vertical direction and the Y axis thereof has moved in the horizontal direction and thereafter the movement deviation of the portable automatic interpretation apparatus50has rapidly decreased is detected, detect this time as the utterance start time.

In this way, if the utterance start time is detected by the utterance start time detection unit12, the utterance start time detection unit12transmits a signal corresponding to the utterance start time to the speech recognition unit20.

At step S14, the speech recognition unit20receives speech data input through the microphone18and temporarily stores the speech data.

At step S16, the utterance end time detection unit14detects an utterance end time. In a state in which the utterance start time is detected and the speech of the user is transferred to and stored in the speech recognition unit20through the microphone18, if the sensor result value of the Y axis of the motion sensing unit10is detected as time at which the portable automatic interpretation apparatus50has moved in the vertical direction and thereafter has less motion deviation, the utterance end time detection unit14recognizes this time as an utterance end time. The final location of the moved portable automatic interpretation apparatus represents a shape similar to that shown inFIG. 8. That is, in the state ofFIG. 7, a portion in which the microphone18is present is moved in the direction of arrow B, and a portion opposite to that of the microphone18is moved in the direction of arrow C.

In this way, if the utterance end time has been detected by the utterance end time detection unit14, the utterance end time detection unit14sends a signal corresponding to the utterance end time to the speech recognition unit20.

At step S18, the speech recognition unit20stops the input of the microphone. As occasion demands, the procedure of step S18may be omitted.

At step S20, the speech recognition unit20converts the speech data stored during a period from the utterance start time to the utterance end time into text, and the translation unit22translates the converted text into text in a target language.

At step S22, the TTS unit24converts the text in the target language output from the translation unit22into speech.

At step S24, the TTS unit24outputs the converted results (speech in the target language) to the outside through the speaker26.

Next, a process for detecting a language pair change command in the embodiment of the present invention will be described in detail with reference to the flowchart ofFIG. 9.

In order for the language pair change detection unit16to detect a language pair change command, a precondition is required. That is, automatic interpretation between a source language and a target language should have been performed at least once in advance. Otherwise, the language pair change detection unit16sensitively reacts to the motion of the portable automatic interpretation apparatus and changes a pair of languages unintended by the user.

At step S30, as described above, after automatic interpretation between the source language and the target language has been performed at least once, the motion sensing unit10continues to sense the motion of the portable automatic interpretation apparatus.

At step S32, the language pair change detection unit16detects a language pair change command based on the output signal of the motion sensing unit10. Here, the movement of the portable automatic interpretation apparatus required to detect a language pair change command must satisfy the following condition. First, the movement of the portable automatic interpretation apparatus must be present in the direction of the Y axis. A situation in which the portable automatic interpretation apparatus (for example, a smart phone) is passed to the other party is taken into consideration, and the other party takes the portable automatic interpretation apparatus and accurately grips the apparatus. In order for the other party to accurately grip the portable automatic interpretation apparatus, inversion between the values of the X axis and the Y axis of the portable automatic interpretation apparatus greatly occurs. The language pair change detection unit16recognizes a signal corresponding to time, at which the portable automatic interpretation apparatus becomes far away from an original device user and the movement of the X and Y axes greatly occurs again and thereafter the movement deviation of the portable automatic interpretation apparatus is decreased, as a language pair change command.

If the language pair change command has been recognized, the language pair change detection unit16sends a signal corresponding to the command to the speech recognition unit20.

At step S34, the speech recognition unit20exchanges the source language and the target language, which have been previously set, for each other.

At step S36, the procedures, such as speech recognition, translation, and TTS conversion shown inFIG. 4, are sequentially performed, and thus speech in the target language is output through the speaker26.

In accordance with the present invention having the above configuration, the portable automatic interpretation apparatus automatically detects speech recognition start time and end time for interpretation, based on the motion of a user's device, thus improving the convenience of the user.

Further, the present invention can more definitely detect the recognition start/end times of the portable automatic interpretation apparatus, thus enhancing the interpretation quality of the portable interpretation apparatus.

Furthermore, the present invention is advantageous in that, in the case of interpretation between two or more users, the change of a language pair can be conducted using only the action of passing the portable automatic interpretation apparatus, so that a source language to be interpreted can be exactly known and the number of manipulations on a screen can be reduced.