Source: http://www.google.com/patents/US6686844?dq=6370535
Timestamp: 2017-09-25 23:04:59
Document Index: 318120194

Matched Legal Cases: ['art 22', 'art 23', 'art 22', 'art 50', 'art 23', 'art 23', 'art 50', 'art 234', 'art 32', 'art 33', 'art 32', 'art 50', 'art 33', 'art 33', 'art 50', 'art 334', 'art 42', 'art 40', 'art 33', 'art 40', 'art 42', 'art 50', 'art 43', 'art 43', 'art 43', 'art 40', 'art 40', 'art 40', 'art 50', 'art 40', 'art 56', 'art 40', 'art 40', 'art 40', 'art 40', 'art 40', 'art 53', 'art 52', 'art 55', 'art 57', 'art 50', 'art 234', 'art 334', 'art 54', 'art 54']

Patent US6686844 - Human interface system using a plurality of sensors - Google Patents
An image sensor, a voice sensor, an auxiliary sensor part (infrared sensor, etc.), a total analyzing part, and an application communicate with each other through data/control signal communication units. Each sensor provides feedback on its signal detection results and control information used by the...http://www.google.com/patents/US6686844?utm_source=gb-gplus-sharePatent US6686844 - Human interface system using a plurality of sensors
Publication number US6686844 B2
Application number US 10/337,818
Also published as US6583723, US20020118232, US20030097190
Publication number 10337818, 337818, US 6686844 B2, US 6686844B2, US-B2-6686844, US6686844 B2, US6686844B2
Inventors Masaki Watanabe, Kentaro Murase, Takuya Noda, Kazuhiro Watanabe
Patent Citations (5), Referenced by (72), Classifications (13), Legal Events (4)
US 6686844 B2
1. A human interface processing method using a plurality of sensors, comprising:
at least two kinds of sensor processings, each determining a range of a detection target and a detection sensitivity and acquiring a particular detection signal from the detection target at the detection sensitivity, the respective detection signals acquired by the sensor processings being of different types;
total analyzing processing, investigating whether or not there is inconsistency among signal detection results detected by the respective sensor processings, and generating control information for the respective sensor processings;
application processing utilizing the signal detection results obtained by the respective sensor processings; and
communication processing communicating data and control information between the respective sensor processings, between the respective sensor processings and the total analyzing processing, and between the total analyzing processing and the application processing, wherein each of the sensor processings uses either of, or a combination of:
the signal detection results or control information obtained from the other sensor processings, and
the control information obtained from the total analyzing processing,
thereby determining a range of a detection target and a detection sensitivity at a time of subsequent signal acquisition;
each of the sensor processings outputs its signal detection results, and control information used by the other sensor processings to determine a range of a detection target and a detection sensitivity at a time of subsequent signal acquisition to the other sensor processings through the communication processing, and
the total analyzing processing outputs control information used by each of the sensor processings to determine a range of a detection target and a detection sensitivity at a time of subsequent signal acquisition through the communication processings.
2. A human interface processing method using a plurality of sensors according to claim 1, wherein the detection target is a human being, and the sensor processings include at least image sensor processing and voice sensor processing,
a detection signal of the image sensor processing is human recognition information,
the image sensor processing includes action recognition processing for interpreting an action of the detection target based on picture recognition results and recognizing a command inputted through a gesture,
a detection signal of the voice sensor processing is human voice recognition information, and
the voice sensor processing includes voice recognition processing for interpreting a voice of the detection target based on voice recognition results and recognizing a command inputted through a voice.
3. A human interface processing method using a plurality of sensors according to claim 1, wherein:
the sensor processings include at least image sensor processing and auxiliary sensor processing;
a detection signal of the image sensor processing is human picture recognition information;
the image sensor processing includes action recognition processing interpreting an action of the detection target based on picture recognition results and recognizing a command inputted through a gesture; and
a detection signal of the auxiliary sensor processing is information useful for detecting human position information.
4. A human interface processing method using a plurality of sensors according 3, wherein one of, or a combination of, ultrasonic sensor processing, infrared sensor processing, and range sensor processing is used as the auxiliary sensor processing.
the sensor processings include at least voice sensor processing and auxiliary sensor processing;
a detection signal of the voice sensor processing inhuman voice recognition information;
the voice sensor processing includes voice recognition processing for interpreting a voice of the detection target based on voice recognition results and recognizing a command inputted through a voice; and
6. A human interface processing method using a plurality of sensors according 5, wherein either of, or a combination of, ultrasonic sensor processing, infrared sensor processing, and range sensor processing is used as the auxiliary sensor processing.
the sensor processings include at least image sensor processing, voice sensor processing, and auxiliary sensor processing;
a detection signal of the voice sensor processing is human voice recognition information;
the voice sensor processing includes voice recognition processing interpreting a voice of the detection target based on voice recognition results and recognizing a command input by a voice input; and
8. A human interface processing method using a plurality of sensors according 7, wherein one of, or a combination of, ultrasonic sensor processing, infrared sensor processing, and range sensor processing is used as the auxiliary sensor processing.
The capture apparatus control part 22 determines the direction, position, and optical parameters for capturing picture information, based on the person detection results by picture analysis of the picture analyzing part 23 a, the sound source detection results by voice analysis of the voice sensor 30 a, and the person position detection results by the auxiliary sensor 40 a, and controls the capture apparatus 21. The capture control part 22 may receive an instruction on the control of capture from the total analyzing part 50 a and the application 60 a.
The picture analyzing part 23 a analyzes a picture. In Embodiment 3, the picture analyzing part 23 a detects a person by picture analysis, and recognizes a command inputted through a gesture.
Hereinafter, action recognition processing in the image sensor 20 a will be described when the image sensor 20 a receives data and feedback of a control instruction from the voice sensor 30 a, the auxiliary sensor 40 a, and the total analyzing part 50 a.
Recognition of a human action based on picture information and voice information is conducted using positional information on a sound source of a person's voice detected by the voice sensor 30 a, and change information in picture of the person at a predetermined time. At this time, the information of the action pattern storage part 234 is referred to.
The recording apparatus control part 32 determines the direction, position, and acoustic parameters for capturing voice information, based on the person detection results of the image sensor 20 a, the sound source detection results of the voice analyzing part 33 a, and the person's position detection results of the auxiliary sensor 40 a, and controls the recording apparatus 31. The recording apparatus control part 32 may receive an instruction on capture control from the total analyzing part 50 a and the application 60 a.
The voice analyzing part 33 a analyzes a voice. In Embodiment 3, the voice analyzing part 33 a detects a person by voice analysis, and recognizes a command inputted through a voice.
Hereinafter, voice recognition processing in the voice sensor 30 a will be described when the voice sensor 30 a receives data and feedback of a control instruction from the image sensor 20 a, the auxiliary sensor 40 a, and the total analyzing part 50 a.
Recognition of a human voice based on picture information and voice information is conducted using object position information having a person's video detected by the image sensor 20 a, and voice information of the person at a predetermined time. At this time, the information from the voice pattern storage part 334 is referred to.
The auxiliary sensor apparatus control part 42 determines the direction, position, and feature parameters for acquiring a signal by the auxiliary sensor part 40 a, based on the person detection results of the image sensor 20 a, the sound source detection results of the voice analyzing part 33 a, and the person's position detection results of the auxiliary sensor part 40 a, and controls the auxiliary sensor apparatus 41. The auxiliary sensor apparatus control part 42 may receive an instruction on the capture control from the total analyzing part 50 a and the application 60 a.
The person's position detecting part 43 is one example of the signal analyzing part 43. The person's position detecting part 43 detects the position of a person based on an output of the auxiliary sensor part 40 a. For example, if the auxiliary sensor apparatus 41 is an ultrasonic sensor, the position of an object is detected by analysis of a reflected wave. If the auxiliary sensor apparatus 41 is an infrared sensor, the position of an object radiating heat at around a human temperature is detected.
Hereinafter, person's position detection processing in the auxiliary sensor part 40 a will be described when the auxiliary sensor part 40 a receives data and feedback of a control instruction from the image sensor 20 a, the voice sensor 30 a, and the total analyzing part 50 a.
Detection of a person's position based on picture information and voice information is conducted using object position information having a person's video detected by the image sensor 20 a, sound source position information detected by/the voice sensor 30 a, and a signal of the person detected by the auxiliary sensor part 40 a at a predetermined time.
The output part 56 provides the person identification results, the person's status recognition results, and the command understanding results to the application 60 a.
The image sensor 20 a may use one or a plurality of capture apparatuses. It is preferable that a capture apparatus, in which the control of optical parameters such as a capture direction, a capture position, a focus, a zoom, and an aperture can be entirely or partly altered, is used as the image sensor 20 a.
The capture direction is determined referring to the results obtained by detecting a person by the auxiliary sensor part 40 a, the results obtained by detecting a sound source direction, using voice information of the voice sensor 30 a, and the results obtained by detecting a person's position based on picture information. More specifically, in the case where the position of a person has been detected prior to a certain point of time; in addition, this position is matched with the estimated sound source direction; and an object is an actual living human being, the capture direction is determined so as to observe the vicinity thereof. However, in the case where the estimated direction of a human being determined immediately before is different from the estimated direction sensed by the auxiliary sensor part 40 a, and in the case where a living human being cannot be confirmed immediately before, the capture direction is determined so as to observe the vicinity of the human detection results of the auxiliary sensor part 40 a.
In this capture direction determination processing, the capture direction is not changed exceeding a speed at which a human being can move. For example, the capture direction is not changed to a position of 5 m away after 0.1 seconds. In the case where the immediately preceding results are different from those of the auxiliary sensor part 40 a, capture is conducted again in the direction in accordance with the immediately preceding results. Furthermore, even in the case where the presence of a living human being has not been confirmed, the last capture direction in which the presence of a human being has been confirmed is compared with the direction pointed by the auxiliary sensor part 40 a. If the capture direction is changed exceeding a moving speed of a human being, the direction in which a picture has been captured last is maintained.
The contents of the command understood as described above are provided to the application 60 a. The information on the position, direction, and posture of a person recognized by the person's status recognizing part 53, the information representing a particular person in the person identifying part 52, and the information on the command understood in the command understanding part 55 are given to the application 60 a, in the case where there is an inquiry from the application 60 a. Alternatively, instead of waiting for an inquiry from the application 60 a, as soon as the information is obtained, it may be sent to the application 60 a via the communication part 57. Alternatively, when the required information is set, it may be sent to the application 60 a.
Next, generation processing of a command dictionary in the total analyzing part 50 a will be described. Prior to the beginning of the application 60 a, a link representing the relation between each command to be accepted by the application 60 a and commands in the action pattern storage part 234 and the voice pattern storage part 334 is stored in the command dictionary storage part 54. Furthermore, a list of commands that can be accepted by the application 60 a in the command dictionary storage part 54 is set by the application 60 in accordance with the situation, after the application 60 a is started. By nullifying the list, commands can be prevented from being accepted.
US4961177 Jan 27, 1989 Oct 2, 1990 Kabushiki Kaisha Toshiba Method and apparatus for inputting a voice through a microphone
JP2000326274A Title not available
JPH01195499A Title not available
US8570330 * Sep 22, 2011 Oct 29, 2013 Sony Corporation Image processing device, image processing method and information storage medium
US20120007893 * Sep 22, 2011 Jan 12, 2012 Sony Computer Entertainment Inc. Image processing device, image processing method and information storage medium
U.S. Classification 340/573.1, 340/686.1, 340/573.7, 340/5.1
International Classification G06T7/20, G06F3/16, G10L15/00, G06T7/00, G06F3/01, G07C9/00
Cooperative Classification G06F3/017, G07C9/00158