Source: https://insight.rpxcorp.com/pat/US6583723B2
Timestamp: 2019-09-22 12:17:06
Document Index: 179500946

Matched Legal Cases: ['art, 50', 'art 40', 'art 40', 'art 40', 'art 40', 'art 50', 'art 40', 'art 50', 'art 50', 'art 50', 'art 40', 'art 50', 'art 50', 'art 50', 'art 40', 'art 50', 'art 40', 'art 40', 'art 40', 'art 50', 'art 50', 'art 50', 'art 40', 'art 50', 'art 50', 'art 50', 'art 40', 'art 50', 'art 50', 'art 50', 'art 40', 'art 50', 'art 40', 'art 22', 'art 23', 'art 24', 'art 23', 'art 231', 'art 232', 'art 233', 'art 234', 'art 22', 'art 23', 'art 22', 'art 50', 'art 23', 'art 23', 'art 50', 'art 234', 'art 234', 'art 50', 'art 50', 'art 234', 'art 32', 'art 33', 'art 34', 'art 33', 'art 331', 'art 332', 'art 333', 'art 334', 'art 32', 'art 33', 'art 32', 'art 50', 'art 33', 'art 33', 'art 50', 'art 334', 'art 334', 'art 50', 'art 50', 'art 334', 'art 42', 'art 43', 'art 44', 'art 42', 'art 40', 'art 33', 'art 40', 'art 42', 'art 50', 'art 43', 'art 43', 'art 43', 'art 40', 'art 40', 'art 40', 'art 50', 'art 40', 'art 40', 'art 50', 'art 40', 'art 50', 'art 40', 'art 50', 'art 51', 'art 52', 'art 53', 'art 54', 'art 55', 'art 56', 'art 57', 'art 52', 'art 40', 'art 52', 'art 40', 'art 51', 'art 53', 'art 40', 'art 54', 'art 234', 'art 334', 'art 56', 'art 40', 'art 50', 'art 40', 'art 23', 'art 33', 'art 43', 'art 40', 'art 40', 'art 50', 'art 40', 'art 40', 'art 40', 'art 232', 'art 234', 'art 40', 'art 332', 'art 334', 'art 50', 'art 52', 'art 51', 'art 55', 'art 54', 'art 40', 'art 40', 'art 40', 'art 40', 'art 40', 'art 40', 'art 40', 'art 40', 'art 40', 'art 40', 'art 40', 'art 40', 'art 40', 'art 40', 'art 232', 'art 234', 'art 40', 'art 232', 'art 232', 'art 40', 'art 40', 'art 234', 'art 234', 'art 40', 'art 40', 'art 40', 'art 40', 'art 40', 'art 40', 'art 40', 'art 50', 'art 51', 'art 54', 'art 234', 'art 334', 'art 52', 'art 50', 'art 52', 'art 52', 'art 40', 'art 53', 'art 40', 'art 40', 'art 54', 'art 53', 'art 52', 'art 55', 'art 57', 'art 60', 'art 234', 'art 334', 'art 54', 'art 54', 'art 50', 'art 50', 'art 51', 'art 52', 'art 55', 'art 53', 'art 52']

Patent US 6,583,723 B2
1. A human interface system using a plurality of sensors, comprising:
at least two kinds of sensors, each determining a range of a detection target and a detection sensitivity and acquiring a particular detection signal from the detection target at the detection sensitivity, the respective detection signals acquired by the sensors being of different types;
a total analyzing part investigating whether or not there is inconsistency among signal detection results obtained by the respective sensors, and generating control information for the respective sensors;
an application utilizing the signal detection results obtained by the respective sensors; and
communication units communicating data and control information between the respective sensors, between the respective sensors and the total analyzing part, and between the total analyzing part and the application, wherein each of the sensors uses one of, or a combination of;
the signal detection results or control information obtained from the other sensors, and the control information obtained from the total analyzing part, thereby determining a range of a detection target and a detection sensitivity at a time of subsequent signal acquisition, each of the sensors outputs its signal detection results and control information used by the other sensors to determine a range of a detection target and a detection sensitivity at a time of subsequent signal acquisition, to the other sensors through the communication units; and
the total analyzing part outputs control information used by each of the sensors to determine a range of a detection target and a detection sensitivity at a time of subsequent signal acquisition through the communication units.
Multi-sensing devices cooperative recognition system
US 7,340,078 B2
US 7,417,551 B2
Ingredient cooking-operation recognition system and computer-readable recording medium which is recorded with ingredient cooking-operation recognition program
US 7,454,390 B2
Ingredient cooking-operation recognition system and ingredient cooking-operation recognition program
US 20070114224A1
US 20060208882A1
US 20050078854A1
Household digital automation control system
US 20050258976A1
TECHCITY TECHNOLOGY CO. LTD.
US 20130302483A1
US 9,389,690 B2
US 9,538,880 B2
Method and apparatus for inputting a voice through a microphone
US 4,961,177 A
2. A human interface system using a plurality of sensors according to claim 1, wherein the detection target is a human being, and the sensors include at least an image sensor and a voice sensor, a detection signal of the image sensor is human picture recognition information, the image sensor includes an action recognizing part interpreting an action of the detection target based on picture recognition results and recognizing a command inputted through a gesture, a detection signal of the voice sensor is human voice recognition information, and the voice sensor includes a voice recognizing part interpreting a voice of the detection target based on voice recognition results and recognizing a command inputted through a voice.
3. A human interface system using a plurality of sensors according to claim 1, wherein:
the detection target is a human being;
the sensors include at least an image sensor and an auxiliary sensor;
a detection signal of the image sensor is human picture recognition information, the image sensor includes an action recognizing part interpreting an action of the detection target based on picture recognition results and recognizing a command inputted through a gesture; and
a detection signal of the auxiliary sensor is information useful for detecting human position information.
4. A human interface system using a plurality of sensors according to claim 3, wherein one of, or a combination of, an ultrasonic sensor, an infrared sensor, and a range sensor is used as the auxiliary sensor.
the sensors include at least a voice sensor and an auxiliary sensor;
a detection signal of the voice sensor is human voice recognition information;
the voice sensor includes a voice recognizing part for interpreting a voice of the detection target based on voice recognition results and recognizing a command inputted through a voice; and
6. A human interface system using a plurality of sensors according to claim 5, wherein either of, or a combination of, an ultrasonic sensor, an infrared sensor, and a range sensor is used as the auxiliary sensor.
the sensors include at least an image sensor, a voice sensor, and an auxiliary sensor, a detection signal of the image sensor is human picture recognition information;
the image sensor includes an action recognizing part interpreting an action of the detection target based on picture recognition results and recognizing a command inputted through a gesture;
a detection signal of the voice sensor is voice recognition information;
the voice sensor includes a voice recognizing part interpreting a voice of the detection target based on voice recognition results and recognizing a command inputted through a voice; and
8. A human interface system using a plurality of sensors according to claim 7, wherein one of, or a combination of, an ultrasonic sensor, an infrared sensor, and a range sensor is used as the auxiliary sensor.
The conventional technique described in JP 2000-32674 A uses capture information from a camera that corresponds to an eye and voice information from a microphone that corresponds to an ear of an apparatus or a robot; however, they are used independently. A block diagram of <FGREF>FIG. 10</FGREF> disclosed in JP 2000-32674 A does not show that information is exchanged between picture information processing and voice information processing. Therefore, the technique described in JP 2000-32674 A has a problem that a picture of a person or a mannequin may be recognized as a human being, and voice information from a loudspeaker of acoustic equipment may be recognized as a human voice. Such matters are not intended by man-machine interface. A picture of a person, a mannequin, and a sound other than a human voice may become a noise for picture recognition and voice recognition, which decreases a recognition ratio. Furthermore, undesired information processing is conducted for inputting picture information and voice information obtained from an undesired target, which decreases a processing speed.
According to the technique described in JP 1(1989)-195499 A, as shown in <FGREF>FIG. 11</FGREF>, positional information on a search target from an ultrasonic sensor and a camera are used for controlling the direction of a microphone; however, processing results of voice information are not used. Furthermore, processing results of voice information from a microphone are not used for position detection control of a search target by an ultrasonic sensor and a camera. According to the technique described in JP 1(1989)-195499 A, in the case where a person enters an area (e.g., a door position of a room) where sensing and capturing are conducted by an ultrasonic sensor and a camera for the purpose of detecting an object, a voice can be efficiently obtained by adjusting the direction of a microphone. However, this is an effective technique only in the case where a narrow search area such as a door position of a room is previously set. Generally, in the case where there is no such limited search area, it may be often assumed that a person stands away from an ultrasonic sensor and a camera, and a command is input through a voice. Thus, the technique described in JP 1(1989)-195499 A cannot flexibly handle such a situation.
<FGREF>FIG. 1</FGREF> is a diagram schematically showing an apparatus configuration of a human interface system using a plurality of sensors of Embodiment 1 according to the present invention.
<FGREF>FIG. 2</FGREF> is a diagram adopting a basic configuration in <FGREF>FIG. 1</FGREF>, in which an exemplary configuration of an image sensor, a voice sensor, and an auxiliary sensor in <FGREF>FIG. 1</FGREF> is shown in detail.
<FGREF>FIG. 3</FGREF> is a flow chart showing a flow of signal processing of the human interface system using a plurality of sensors of Embodiment 1 according to the present invention.
<FGREF>FIG. 4</FGREF> is a flow chart showing a flow of signal processing of a total analyzing part in Embodiment 1 according to the present invention.
<FGREF>FIGS. 5A</FGREF> to 5C are diagrams showing exemplary configurations of Embodiment 2 according to the present invention, in which the configuration of Embodiment 1 is modified.
<FGREF>FIG. 6</FGREF> is a block diagram schematically showing an apparatus configuration of a human interface system using a plurality of sensors of Embodiment 3 according to the present invention.
<FGREF>FIG. 7</FGREF> is a flow chart showing a processing flow of an entire human interface system of Embodiment 3 according to the present invention.
<FGREF>FIG. 8</FGREF> is a block diagram schematically showing an apparatus configuration of a human interface system using a plurality of sensors of Embodiment 4 according to the present invention.
<FGREF>FIG. 9</FGREF> is a diagram showing recording media storing a processing program for realizing a human interface system using a plurality of sensors of Embodiment 5 according to the present invention.
<FGREF>FIG. 10</FGREF> is a diagram showing an example of a conventional interface using sensors.
<FGREF>FIG. 11</FGREF> is a diagram showing another example of a conventional interface using sensors.
<FGREF>FIG. 1</FGREF> shows a schematic apparatus configuration of the human interface system using a plurality of sensors of Embodiment 1 according to the present invention. Reference numeral 10 denotes a sensor part, which includes an image sensor 20 and a voice sensor 30. Reference numeral 40 denotes an auxiliary sensor part, 50 denotes a total analyzing part, and 60 denotes an application. Herein, the auxiliary sensor part 40 includes at least one sensor capable of acquiring a useful signal other than those acquired by the image sensor 20 and the voice sensor 30, in accordance with a target for which the human interface system of the present invention is used.
As shown in <FGREF>FIG. 1</FGREF>, in the human interface system using a plurality of sensors of Embodiment 1 according to the present invention, the image sensor 20 and the voice sensor 30, and the auxiliary sensor part 40 respectively use signal processing results obtained by the other sensors during signal processing, and give an instruction on the control of the other sensors. As shown in <FGREF>FIG. 1</FGREF>, a data/control signal communication unit 71 is provided for bidirectionally exchanging signal processing results and control data between the image sensor 20 and the voice sensor 30. A data/control signal communication unit 72 is also provided for bidirectionally exchanging signal processing results and control data between the image sensor 20 and the auxiliary sensor part 40. A data/control signal communication unit 70 is also provided for bidirectionally exchanging signal processing results and control data between the voice sensor 30 and the auxiliary sensor part 40. Furthermore, data/control signal communication units 73, 74, and 75 are provided for bidirectionally exchanging signal processing results and control data between the image sensor 20 and the total analyzing part 50, between the auxiliary sensor part 40 and the total analyzing part 50, and between the voice sensor 30 and the total analyzing part 50. A data/control signal communication unit 76 is also provided for bidirectionally exchanging signal processing results and control data between the total analyzing part 50 and the application 60.
According to the configuration shown in <FGREF>FIG. 10</FGREF> described in the prior art, signal processing in each sensor is independently conducted, and the processing results are given to an application as independent information. According to the configuration in <FGREF>FIG. 11</FGREF>, although the processing results of the ultrasonic sensor and the image sensor are used in the processing of the voice sensor, information and control flow in one direction. In contrast, according to the present invention, the image sensor 20, the voice sensor 30, and the auxiliary sensor part 40 use the processing results of the other sensors through the data/control signal communication units 70, 71 and 72, and give an instruction on the control of the other sensors. Furthermore, each sensor uses the processing results of the total analyzing part 50 by feedback through the data/control signal communication units 73, 74, and 75, and receives an instruction on the control of each sensor from the total analyzing part 50. The total analyzing part 50 receives all the sensor processing results of the image sensor 20, the voice sensor 30, and the auxiliary sensor part 40, and conduct feedback processing while investigating the consistency among the sensors, thereby generating an output with consistency in the entire apparatus. Furthermore, the total analyzing part 50 refers to the execution results of the application 60 through the data/control signal communication unit 76, in accordance with the situation, receives an instruction on signal processing from the application 60, and controls each sensor based on the instruction.
<FGREF>FIG. 2</FGREF> adopts the basic configuration in <FGREF>FIG. 1</FGREF>, in which an exemplary configuration of the image sensor 20, the voice sensor 30, and the auxiliary sensor 40 are described in detail.
A human interface system using a plurality of sensors of Embodiment 2 according to the present invention is obtained by modifying the configuration in Embodiment 1. In the configuration shown in <FGREF>FIG. 1</FGREF> described in Embodiment 1, there are provided the image sensor 20, the voice sensor 30, and the auxiliary sensor part 40. <FGREF>FIGS. 6A</FGREF> to 5C show variations of the configuration in Embodiment 1. <FGREF>FIG. 5A</FGREF> shows the configuration shown in <FGREF>FIG. 1</FGREF> with the auxiliary sensor part 40 omitted. <FGREF>FIG. 5B</FGREF> shows the configuration shown in <FGREF>FIG. 1</FGREF> with the voice sensor 30 omitted. <FGREF>FIG. 5C</FGREF> shows the configuration shown in <FGREF>FIG. 1</FGREF> with the image sensor 20 omitted.
<FGREF>FIG. 5A</FGREF> shows the case where the auxiliary sensor part 40 is not required. Signal processing results are obtained between the image sensor 20 and the voice sensor 30 through the data/control signal communication unit 71. Furthermore, the image sensor obtains the analysis results from the total analyzing part 50 through the data/control signal communication unit 73. Furthermore, the voice sensor obtains the analysis results from the total analyzing part 50 through the data/control signal communication unit 75. Furthermore, the contents of the control instruction from the application 50 is obtained via the total analyzing part 50 through the data/control signal communication unit 76. Among all the results, those with the least inconsistency among a plurality of sensors are obtained.
<FGREF>FIG. 5B</FGREF> shows the case where the voice sensor 30 is omitted, and an application is operated only with picture information. Signal processing results are obtained between the image sensor 20 and the auxiliary sensor part 40 through the data/control signal communication unit 72. Furthermore, the auxiliary sensor part obtains the analysis results from the total analyzing part 50 through the data/control signal communication unit 74. Furthermore, the image sensor obtains the analysis results from the total analyzing part 50 through the data/control signal communication unit 73. Furthermore, the contents of the control instruction from the application 60 is obtained via the total analyzing part 50 through the data/control signal communication unit 76. Among all the results, those with the least inconsistency are obtained.
<FGREF>FIG. 5C</FGREF> shows the case where the image sensor 20 is omitted, and an application is operated only with voice information. Signal processing results are obtained between the voice sensor 30 and the auxiliary sensor part 40 through the data/control signal communication unit 72. Furthermore, the auxiliary sensor part obtains the analysis results from the total analyzing part 50 through the data/control signal communication unit 74. Furthermore, the voice sensor obtains the analysis results from the total analyzing part 50 through the data/control signal communication unit 75. Furthermore, the contents of the control instruction from the application 60 is obtained via the total analyzing part 50 through the data/control signal communication unit 76. Among all the results, those with the least inconsistency are obtained.
<FGREF>FIG. 6</FGREF> is a block diagram showing a schematic configuration of an apparatus of the human interface system using a plurality of sensors of Embodiment 3 according to the present invention. As shown in <FGREF>FIG. 6</FGREF>, the apparatus includes an image sensor 20a, a voice sensor 30a, an auxiliary sensor part 40a, a total analyzing part 50a, and an application 60a. Detection targets are a human being, a human voice, and a human gesture.
Herein, the image sensor 20a includes a gesture analyzing function for analyzing the contents of an operation instruction based on a human movement and a gesture, in addition to the function of obtaining human picture data. Furthermore, the voice sensor 30a includes a voice analyzing function for analyzing the contents of an operation instruction based on a human voice, in addition to the function of obtaining human voice data. Furthermore, the auxiliary sensor part 40a acquires a useful signal with respect to the processing of the image sensor 20a and the voice sensor 30a. For example, an ultrasonic sensor is a sensor useful for detecting the position of a human being and that a human being is not a still object such as a mannequin and a radio, but a moving object. An infrared sensor (i.e., a sensor generating thermography) is a sensor useful for detecting that a human being is not a lifeless object such as a mannequin and a radio, but a living creature.
The image sensor 20a includes a capture apparatus 21, a capture apparatus control part 22, a picture analyzing part 23a, and a communication part 24. The picture analyzing part 23a includes a person detecting part 231, a person's video storage part 232, an action recognizing part 233, and an action pattern storage part 234.
The capture apparatus control part 22 determines the direction, position, and optical parameters for capturing picture information, based on the person detection results by picture analysis of the picture analyzing part 23a, the sound source detection results by voice analysis of the voice sensor 30a, and the person position detection results by the auxiliary sensor 40a, and controls the capture apparatus 21. The capture control part 22 may receive an instruction on the control of capture from the total analyzing part 50a and the application 60a.
The picture analyzing part 23a analyzes a picture. In Embodiment 3, the picture analyzing part 23a detects a person by picture analysis, and recognizes a command inputted through a gesture.
Hereinafter, action recognition processing in the image sensor 20a will be described when the image sensor 20a receives data and feedback of a control instruction from the voice sensor 30a, the auxiliary sensor 40a, and the total analyzing part 50a.
Recognition of a human action based on picture information and voice information is conducted using positional information on a sound source of a person's voice detected by the voice sensor 30a, and change information in picture of the person at a predetermined time. At this time, the information of the action pattern storage part 234 is referred to.
Recognition of a human action based on picture information, voice information, and person detection information from the auxiliary sensor 40a is conducted using positional information on a sound source of a person's voice detected by the voice sensor 30a, person's position information detected by the auxiliary sensor 40a, and change information in picture of the person at a predetermined time. At this time, the information of the action pattern storage part 234 is referred to.
Recognition of a human action based on picture information, voice information, human detection information from the auxiliary sensor 40a, analysis results of the total analyzing part 50a, and a control instruction from the application 60a is conducted using positional information on a sound source of a person's voice detected by the voice sensor 30a, the person's position information detected by the auxiliary sensor 40a, analysis information with the least inconsistency among the sensors obtained by the total analyzing part 50a, the contents of a control instruction from the application 60a, and change information in picture of the person at a predetermined time. At this time, the information from the action pattern storage part 234 is referred to.
Next, the voice sensor 30a includes a recording apparatus 31, a recording apparatus control part 32, a voice analyzing part 33a, and a communication part 34. The voice analyzing part 33a includes a sound source detecting part 331, a person's voice storage part 332, a voice recognizing part 333, and a voice pattern storage part 334.
The recording apparatus control part 32 determines the direction, position, and acoustic parameters for capturing voice information, based on the person detection results of the image sensor 20a, the sound source detection results of the voice analyzing part 33a, and the person's position detection results of the auxiliary sensor 40a, and controls the recording apparatus 31. The recording apparatus control part 32 may receive an instruction on capture control from the total analyzing part 50a and the application 60a.
The voice analyzing part 33a analyzes a voice. In Embodiment 3, the voice analyzing part 33a detects a person by voice analysis, and recognizes a command inputted through a voice.
Hereinafter, voice recognition processing in the voice sensor 30a will be described when the voice sensor 30a receives data and feedback of a control instruction from the image sensor 20a, the auxiliary sensor 40a, and the total analyzing part 50a.
Recognition of a human voice based on picture information and voice information is conducted using object position information having a person's video detected by the image sensor 20a, and voice information of the person at a predetermined time. At this time, the information from the voice pattern storage part 334 is referred to.
Recognition of a human voice based on picture information, voice information, and person detection information from the auxiliary sensor 40a is conducted using object position information having a person's video detected by the image sensor 20a, person's position information detected by the auxiliary sensor 40a, and voice information of the person at a predetermined time. At this time, the information from the voice pattern storage part 334 is referred to.
Recognition of a human voice based on picture information, voice information, human detection information from the auxiliary sensor 40a, analysis results of the total analyzing part 50a, and a control instruction of the application 60a is conducted using object position information having a person's video detected by the image sensor 20a, human position information detected by the auxiliary sensor 40a, analysis information with the least inconsistency among the sensors obtained by the total analyzing part 50a, the contents of a control instruction from the application 60a, and voice information of the person at a predetermined time. At this time, the information from the voice pattern storage part 334 is referred to.
The auxiliary sensor 40a includes an auxiliary sensor apparatus 41, an auxiliary sensor apparatus control part 42, a person's position detecting part 43, and a communication part 44.
The auxiliary sensor apparatus control part 42 determines the direction, position, and feature parameters for acquiring a signal by the auxiliary sensor part 40a, based on the person detection results of the image sensor 20a, the sound source detection results of the voice analyzing part 33a, and the person's position detection results of the auxiliary sensor part 40a, and controls the auxiliary sensor apparatus 41. The auxiliary sensor apparatus control part 42 may receive an instruction on the capture control from the total analyzing part 50a and the application 60a.
The person's position detecting part 43 is one example of the signal analyzing part 43. The person's position detecting part 43 detects the position of a person based on an output of the auxiliary sensor part 40a. For example, if the auxiliary sensor apparatus 41 is an ultrasonic sensor, the position of an object is detected by analysis of a reflected wave. If the auxiliary sensor apparatus 41 is an infrared sensor, the position of an object radiating heat at around a human temperature is detected.
Hereinafter, person's position detection processing in the auxiliary sensor part 40a will be described when the auxiliary sensor part 40a receives data and feedback of a control instruction from the image sensor 20a, the voice sensor 30a, and the total analyzing part 50a.
Detection of a person's position based on picture information and voice information is conducted using object position information having a person's video detected by the image sensor 20a, sound source position information detected by the voice sensor 30a, and a signal of the person detected by the auxiliary sensor part 40a at a predetermined time.
Voice recognition based on picture information, voice information, person detection information from the auxiliary sensor part 40a, analysis results of the total analyzing part 50a, and a control instruction from the application 60 is conducted using object position information having a person's video detected by the image sensor 20a, sound source position information detected by the voice sensor 30a, human position information detected by the auxiliary sensor part 40a, analysis information without inconsistency among the sensors obtained by the total analyzing part 50a, the contents of a control instruction from the application 60, and a signal of the person detected by the auxiliary sensor part 40a at a predetermined time.
The total analyzing part 50a includes a person's information storage part 51, a person identifying part 52, a person's status recognizing part 53, a command dictionary storage part 54, a command understanding part 55, an output part 56, and a communication part 57.
The person identifying part 52 identifies a person detected by the image sensor 20a, the voice sensor 30a, and the auxiliary sensor part 40a. In identification processing, the person identifying part 52 studies the identification results of the image sensor 20a, those of the voice sensor 30a, those of the auxiliary sensor part 40a, and the information from the person's information storage part 51 in a comprehensive manner, and in the case where results with inconsistency are obtained, determines any or all of the identification results of the respective sensors to be incorrect.
The person's status recognizing part 53 determines the position, direction, and posture of a person, based on a person's position calculated by using the information from the auxiliary sensor part 40a, human detection results based on picture information, and sound source detection results based on voice information.
The command dictionary storage part 54 stores two kinds of information. One is a link representing a relation between each command given to the application 60a and a command in the action pattern storage part 234 and the voice pattern storage part 334. The other is a list of commands that can be accepted by the application in the respective cases. The latter information is rewritten by the application 60a during execution from one minute to another.
The output part 56 provides the person identification results, the person's status recognition results, and the command understanding results to the application 60a.
Next, a processing flow is described.
<FGREF>FIG. 7</FGREF> is a flow chart showing an outline processing flow in the human interface system of Embodiment 3. Each process will be described in detail later.
First, each of the image sensor 20a, the voice sensor 30a, and the auxiliary sensor part 40a are activated in parallel with the total analyzing part 50 (Operation 701).
Each of the image sensor 20a, the voice sensor 30a, and the auxiliary sensor part 40a analyzes the signal acquired in Operation 702, using the picture analyzing part 23a, the voice analyzing part 33a, or the signal analyzing part 43, and outputs analysis results (Operation 703).
Each of the image sensor 20a, the voice sensor 30a, and the auxiliary sensor part 40a obtains analysis results of the other sensors in the image sensor 20a, the voice sensor 30a, and the auxiliary sensor part 40a, and the total analyzing part 50 through the data/control signal communication units 70 to 75 (Operation 704).
Next, the auxiliary sensor part 40a is controlled referring to the obtained analysis results to obtain information, and person's position detection processing is conducted based on the information in the auxiliary sensor part 40a. First, it is checked whether or not person's position detection processing can be conducted with the currently acquired signal (Operation 705). In the case where the processing can be conducted (Operation 705: Y), person's position detection processing is conducted by using the currently acquired signal to output the results (Operation 706).
Next, the image sensor 20a controls the capture apparatus 21 to obtain a picture, referring to the output results of the auxiliary sensor part 40a, the voice processing results of the voice sensor 30a, and the picture processing results, checks its color information to determine the position of a person, and recognizes a human action. First, it is checked whether or not person's action recognition processing can be performed with the currently acquired signal (Operation 709). In the case where the processing can be conducted (Operation 709: Y), person's action recognition processing is conducted with the currently acquired signal to output the results (Operation 710). At this time, video data previously registered in the person's video storage part 232 and information on an action pattern in the action pattern storage part 234 are referred to.
Next, the voice sensor 30a controls the recording apparatus 31 referring to the output results of the auxiliary sensor part 40a, the picture processing results of the image sensor 20a, and the voice processing results to obtain a voice. Then, the voice sensor 30a checks its wavelength information to determine the direction of a sound source, and recognizes a human voice. First, it is checked whether or not person's voice recognition processing can be conducted with the currently acquired signal (Operation 713). In the case where the processing can be conducted (Operation 713: Y), person's voice recognition processing is conducted with the currently acquired signal to output the results (Operation 714). At this time, voice data previously registered in the person's voice storage part 332 and information on a voice pattern in the voice pattern storage part 334 are referred to.
Next, in the total analyzing part 50a, person's status recognition, person identification, and command understanding are conducted based on picture information, voice information, and information from the auxiliary sensor (Operation 717). In person identification, the person identifying part 52 identifies a person, referring to video information and voice information of a particular person previously registered in the person's information storage part 51. In command understanding, the command understanding part 55 refers to the information from the command dictionary storage part 54.
First, processing in the auxiliary sensor part 40a will be described in detail.
An example in which the capture apparatus and the recording apparatus utilize the information from the auxiliary sensor part 40a will be described. In the case of using an ultrasonic sensor, the positional information on a moving object in a space can be provided to the capture apparatus and the recording apparatus. When the positional information is known, the capture apparatus may detect a person only in a limited region in a large space, which enables a processing time to be substantially reduced. Furthermore, when the positional information is known, the recording apparatus does not need to detect the position of a sound source.
Next, an example in which the auxiliary sensor part 40a utilizes information from the capture apparatus and the recording apparatus will be described. In the case where the capture apparatus 21 and the recording apparatus 31 detects the position of a person and the position of a sound source by the function of each apparatus irrespective of the positional information from an ultrasonic sensor, the ultrasonic sensor scans the vicinity of the position based on the positional information and detects the presence/absence of a moving object (person). Without an auxiliary sensor, the picture apparatus may recognize a portion other than a human being as the position of a human being, and the recording apparatus may detect a sound from a non-human sound source such as a TV and audio equipment. The ultrasonic sensor has an effect of preventing such erroneous detection.
Furthermore, an example in which the auxiliary sensor part 40a utilizes positional information on a moving object in a space to be detected by the auxiliary sensor part 40a. The auxiliary sensor part 40a utilizes positional information on a known moving object to restrict scanning only to the vicinity of the object, whereby high-speed scanning is realized. The auxiliary sensor part 40a can dynamically change a scanning position so as to follow the movement of the object. Thus, the auxiliary sensor part 40a is effective as an active sensor for tracking an object in real time.
Next, processing centering on the image sensor 20a will be described in detail.
The image sensor 20a may use one or a plurality of capture apparatuses. It is preferable that a capture apparatus, in which the control of optical parameters such as a capture direction, a capture position, a focus, a zoom, and an aperture can be entirely or partly altered, is used as the image sensor 20a.
In the case where a distance as well as a direction of a human place from the capture apparatus 21 are desired to be obtained in person's detection based on picture information, a plurality of capture apparatuses are used. Alternatively, one capture apparatus can be used if a capture position can be altered. Even if a capture position cannot be altered, optical parameters such as a focus, a zoom, and an aperture may be altered. In any apparatus setting, an apparatus/apparatuses is disposed so that at least two pictures can be captured depending upon different capture positions or different optical parameters, with respect to a human being in a range from which an application desires to receive a command. If the above-mentioned condition is satisfied, it is also possible to use a capture apparatus using a fish-eye lens, a capture apparatus capable of obtaining an arbitrary pixel of a picture by a CMOS system, and a capture apparatus capable of obtaining color other than visible light.
The capture direction is determined referring to the results obtained by detecting a person by the auxiliary sensor part 40a, the results obtained by detecting a sound source direction, using voice information of the voice sensor 30a, and the results obtained by detecting a person's position based on picture information. More specifically, in the case where the position of a person has been detected prior to a certain point of time; in addition, this position is matched with the estimated sound source direction; and an object is an actual living human being, the capture direction is determined so as to observe the vicinity thereof. However, in the case where the estimated direction of a human being determined immediately before is different from the estimated direction sensed by the auxiliary sensor part 40a, and in the case where a living human being cannot be confirmed immediately before, the capture direction is determined so as to observe the vicinity of the human detection results of the auxiliary sensor part 40a.
In this capture direction determination processing, the capture direction is not changed exceeding a speed at which a human being can move. For example, the capture direction is not changed to a position of 5 m away after 0.1 seconds. In the case where the immediately preceding results are different from those of the auxiliary sensor part 40a, capture is conducted again in the direction in accordance with the immediately preceding results. Furthermore, even in the case where the presence of a living human being has not been confirmed, the last capture direction in which the presence of a human being has been confirmed is compared with the direction pointed by the auxiliary sensor part 40a. If the capture direction is changed exceeding a moving speed of a human being, the direction in which a picture has been captured last is maintained.
Optical parameters such as a focus, a zoom, and an aperture in capturing are adjusted so as to capture the vicinity of the position of a human being detected immediately before. In the case where the auxiliary sensor part 40a can output a distance to a human being, such voice information and picture information are referred to in the same way as in the control of a direction.
In person detection based on an image, sound source detection based on a voice, and person's position detection by the auxiliary sensor 40a, when a person and a sound source have not been detected due to the influence of an obstacle, the direction and position in which a picture should be captured are specified from each detection process. Therefore, the capture apparatus 21 is controlled so as to observe the specified direction and position.
The person's video storage part 232 stores person's video pattern information that allows a human being to be recognized and the position and direction of the human being to be specified. The direction of a human being in a video pattern is assumed to be a capture direction of a human being in human recognition and position and direction specification. This is based on the position of the previously disposed capture apparatus 21, and a range of a human direction with respect to a display apparatus displaying an output of the application 60a and a display apparatus whose operation is permitted by the application 60a. A resolution (which may be a “size”, corresponding to an area converted by the number of pixels) of a picture for capturing a human being is assumed to be a resolution required for specifying the direction of a human being. In the case where the direction of a human being is determined by the direction of a face, a picture with a resolution that allows the position of a contour of a face, eye, a nose, and a mouth to be determined. In the case where the direction of a human being is determined based on the direction of eyes, a picture is captured so that eyes can be identified, and the positions of white parts and black parts of the eyes are determined so as to identify a line of sight.
As described above, a plurality of pictures of a human being may be captured in accordance with the directions of the capture apparatus and the human being and the distance therebetween. Furthermore, in accordance with the requirements of the application 60a, pictures with different facial expressions (smiling face, crying face, etc.), and pictures with and without make-up, beard, and glasses may be captured.
The action pattern storage part 234 stores the action of a human being required for the application 60a to recognize a command. Changes in coordinates of the skin in a main joint portion, such as a hand, a leg, a head, and an elbow can also be stored as an action pattern.
Next, person detection processing based on picture information in the image sensor 20a will be described. The person detection processing is conducted as follows.
Picture information used for detecting a human being, captured by the image sensor 20a, is assumed to be captured by referring to the output of the auxiliary sensor part 40a, the sound source detection results of the voice sensor 30a, and the human being detection results obtained by the image sensor 20a. Color information and positional information of pixels are used for searching for a region corresponding to a face in the input picture. Then, the direction in which a human being is present is calculated from the position of the capture apparatus and the detection position on the picture in accordance with a camera model such as a pinhole camera. In the case where there are a plurality of capture apparatuses and a human being is detected on a plurality of pictures, the position as well as the direction of a human being is also calculated based on the principle of triangulation.
As picture information representing a human face, those registered in the person's video storage part 232 are used. As described later, those registered in the person's video storage part 232 include a person's picture captured from the front and a person's picture captured from the side in accordance with the specification of the application 60a. In this case, person's face is detected in all the registered directions.
In the case where the presence of a human being cannot be confirmed even if it was confirmed immediately before, each sensor is instructed to confirm information by enhancing the sensitivity. More specifically, the recording apparatus is instructed to increase a volume and a sampling interval. The auxiliary sensor part 40a is instructed to decrease a scanning interval of an environment. The capture apparatus is instructed to enhance a resolution by a zoom. In the case where it is determined that a person cannot be detected, it is determined that there is not a human being in a capture direction.
Action recognition based on picture information and voice information in the image sensor 20a is conducted as follows. Picture information used for action recognition is assumed to be captured by referring to the output of the auxiliary sensor part 40a, the sound source detection results of the voice sensor 20a, and the immediately preceding human detection results of the image sensor 20a. In the same way as in the information stored in the above-mentioned action pattern storage part 234, information is extracted from an input picture, and its change is matched with an action pattern stored in the action pattern storage part 234. Matching between the action in the input picture and the action in the action pattern is determined by obtaining the difference between corresponding pictures in both continuous pictures as a moving picture, and accumulating the difference values throughout the moving picture. Alternatively, a change in the input picture is approximated by discrete cosine transformation (DCT), Fourier transformation, or the like, and the action in the similarly approximated action pattern is compared with a transformed coefficient, whereby matching is determined.
In the case where the presence of a human being is confirmed, but an action command cannot be understood, each sensor is instructed to confirm information by enhancing the sensitivity. More specifically, the recording apparatus is instructed to increase a recording volume and a sampling interval. The auxiliary sensor part 40a is instructed to decrease a scanning interval of an environment. The capture apparatus is instructed to enhance a resolution by a zoom.
Next, processing centering on the voice sensor 30a will be described in detail.
The voice sensor 30a is disposed in such a manner that a plurality of recording apparatuses or one recording apparatus can obtain a human voice in a range from which the application 60a is to receive a command.
The output of the recording apparatus is assumed to be a signal regarding a voice in a particular direction, referring to the human detection results obtained by the auxiliary sensor part 40a, the results of detecting a human position obtained by using picture information, and the results of sound source direction detected based on voice information. More specifically, in the case where a sound corresponding to a human voice is detected before a certain point of time, the direction of the detected voice is matched with the direction determined from a picture in which a human being is present, and an object is identified as an actual human being, information on a voice signal from the vicinity of the direction is output. However, in the case where the direction in which the presence of a human being is determined immediately before is different from the direction in which the presence of a human being is determined by the auxiliary sensor 40a, or in the case where a human being cannot be confirmed immediately before, information on a voice signal from the vicinity of the direction of the person detection results obtained by the auxiliary sensor part 40a is output.
In the case where a human being and a sound source cannot be detected due to the influence of an obstacle in human detection based on a picture, a sound source detection based on a voice, and person's position detection by the auxiliary sensor 40a, the direction in which the recording apparatus 31 is to record is specified from each detection process. Therefore, voice information from the direction is output.
Next, sound source detection processing will be described. Voice information used for sound source detection is assumed to be voice information output in a particular direction, referring to the output of the auxiliary sensor part 40a, the human detection results of the image sensor 20a, and the immediately preceding sound source detection results. The presence/absence of a person's voice is determined by investigating frequency characteristics of voice information and determining matching between the frequency characteristics of voice information and the information stored as a person's voice described later. In the case where it is determined that there is no person's voice, it is determined that there is no human being in a direction in which voice information is obtained.
In the case where a plurality of recording apparatuses 31 are disposed at some distance from each other, a sound source position is specified based on the difference in time at which a voice is recorded. Furthermore, in the case where a plurality of recording apparatuses are disposed respectively at a plurality of places, a sound source direction is specified at each place. An overlapped sound source direction calculated from a group of recording apparatuses at the respective places is obtained, whereby a sound source position is determined. In the case where a sound source cannot be confirmed even if a human voice was confirmed immediately before, each sensor is instructed to confirm information by enhancing the sensitivity. More specifically, the recording apparatus is instructed to increase a recording volume and a sampling interval. The auxiliary sensor part 40a is instructed to decrease a scanning interval of an environment. The capture apparatus is instructed to enhance a resolution by a zoom.
In voice recognition based on voice information and picture information, picture information used for the voice recognition is assumed to be captured by referring to the output of the auxiliary sensor part 40a, the sound source detection results, and the immediately preceding person detection results. In the case where a human voice is detected, but a voice command cannot be understood, each sensor is instructed to confirm information by enhancing the sensitivity. More specifically, the recording apparatus is instructed to increase a recording volume and a sampling interval. The auxiliary sensor part 40a is instructed to decrease a scanning interval of an environment. The capture apparatus is instructed to enhance a resolution by a zoom.
Next, processing centering on the total analyzing part 50a will be described in detail.
The person's information storage part 51 stores video information, voice information, and information required for identifying a particular person or a plurality of persons which the application 60a operates. As the video information, a sample video of each person required for specifying a person is stored. A plurality of pictures may be previously captured in plural directions and at plural distances. Furthermore, depending upon the requirements of the application 60a, pictures with different facial expressions (smiling face, crying face, etc.), pictures with and without make-up, beard, and glasses may be captured. As long as a picture required for specifying a person can be reproduced, the amount of data of a picture to be stored may be compressed. Furthermore, only a face or only a part such as eyes, a mouth, and a nose required for confirming the difference between individuals may be stored. Furthermore, the above-mentioned information may be combined. Alternatively, as long as the information can be reproduced, the information is compressed, and stored in combination.
As the voice information, a voice spoken by each person required for specifying a person is stored. A spoken voice of each person is stored individually. A command to be received by the application 60a is stored, or a spoken voice required for reproducing a command to be received by the application 60a is stored. In the latter case, a phoneme and a word obtained by decomposing a command is recorded. In storage, frequency characteristics for each command, phoneme, or word, or changes in frequency characteristics can be stored instead of that a recorded voice is stored as it is. The above-mentioned information may be combined. Alternatively, as long as the information can be reproduced, it can be compressed, and stored in combination.
The command dictionary storage part 54 stores two kinds of information. One is a link representing a relation between each command inputted to the application 60a and a command in the action pattern storage part 234 and the voice pattern storage part 334. The other is a list of commands that can be accepted by the application 60a in the respective cases. The latter command dictionary is rewritten by the application 60a successively in accordance with the process of the application 60a. Thus, in command understanding processing based on action recognition results and voice recognition results, a command dictionary is always referred to in each processing.
Next, person identification in the person identifying part 52 of the total analyzing part 50a will be described. The person identifying part 52 can confirm a person's video in person detection processing in the image sensor 20a and detect a person's voice in sound source detection processing in the voice sensor 30a. Only in the case where the position detected by the person detection results and the position detected by the sound source detection results are overlapped, it is determined that a person can be observed correctly. In person identification and command understanding described later, the picture information and voice information are treated as effective ones only when an existence of a human being is confirmed. In the case where the person identifying part 52 determines that a person is observed, the position and direction of a human being are output as the position where the person position detection results of the auxiliary sensor part 40a, the person detection results based on picture information, and the sound source detection results based on voice information are overlapped. A posture follows the person detection results based on the picture information.
Next, person's status recognition processing in the person's status recognition part 53 will be described. Herein, a person's status refers to the position, direction, and posture of a person. A direction in which a human being is present is found from the auxiliary sensor part 40a. A distance from the apparatus to the human being is also found by the function of the auxiliary sensor part 40a. Only in the case where the presence of a human being is confirmed, a direction and a distance are found from the picture information of the image sensor 20a. Furthermore, in the case where information on the direction of a face is registered, a posture (direction of a face) is also found. From the voice information of the voice sensor 30a, the direction in which a human voice is heard is found only in the case where a human voice is confirmed. In the case where a plurality of recording apparatuses are disposed at a plurality of places in an environment, the position of a human being in the environment is found.
Correspondence between the commands is checked. One is the command obtained by the action recognition results and the voice recognition results while a person is identified in person identification. The other is the command of the application 60a stored in the command dictionary storage part 54. In the case where the command determined from an action and the command determined from a voice correspond to the commands of the application 60a, and the commands are in a list of commands which the application 60a can accept, these commands are determined to be understood. In the case where it is determined that a person is identified in person identification, and one of the action recognition results and the voice recognition results is obtained, if the command obtained from the results is in a list of commands which the application 60a can accept, this command is determined to be understood.
The contents of the command understood as described above are provided to the application 60a. The information on the position, direction, and posture of a person recognized by the person's status recognizing part 53, the information representing a particular person in the person identifying part 52, and the information on the command understood in the command understanding part 55 are given to the application 60a, in the case where there is an inquiry from the application 60a. Alternatively, instead of waiting for an inquiry from the application 60a, as soon as the information is obtained, it may be sent to the application 60a via the communication part 57. Alternatively, when the required information is set, it may be sent to the application 60a.
Next, generation processing of a command dictionary in the total analyzing part 60a will be described. Prior to the beginning of the application 60a, a link representing the relation between each command to be accepted by the application 60a and commands in the action pattern storage part 234 and the voice pattern storage part 334 is stored in the command dictionary storage part 54. Furthermore, a list of commands that can be accepted by the application 60a in the command dictionary storage part 54 is set by the application 60 in accordance with the situation, after the application 60a is started. By nullifying the list, commands can be prevented from being accepted.
<FGREF>FIG. 8</FGREF> shows an exemplary configuration of Embodiment 4.
In the exemplary configuration of Embodiment 4 shown in <FGREF>FIG. 8</FGREF>, an action command, a voice command, and a person's status of an unspecified person. Compared with the exemplary configuration shown in <FGREF>FIG. 6</FGREF> described in Embodiment 3, the configuration of the total analyzing part 50b is different. In the configuration of Embodiment 4, the total analyzing part 50b does not include a person's information storage part 51 and a person identifying part 52. Furthermore, the command understanding part 55 receives the results of the person's status recognizing part 53 directly, not via the person identifing part 52.
Watanabe, Kazuhiro, Watanabe, Masaki, Murase, Kentaro, Noda, Takuya
US 20020118232A1
340/573.1, 340/517, 340/523, 340/407.1, 340/5.83, 340/5.84, 340/5.81, 345/863, 345/978, 345/328, 345/208, 345/358, 345/184, 345/522, 345/520, 345/104, 345/107, 345/156