Source: http://www.google.com/patents/US6686844?dq=7,479,949
Timestamp: 2016-05-01 05:09:21
Document Index: 16183655

Matched Legal Cases: ['art 50', 'art 50', 'art 50', 'art 50', 'art 40', 'art 40', 'art 333', 'art 31', 'art 334', 'art 40', 'art 40', 'art 50', 'art 40', 'art 40', 'art 53', 'art 52', 'art 55', 'art 57', 'art 50', 'art 234', 'art 334', 'art 54', 'art 54', 'art 50', 'art 50', 'art 51', 'art 52', 'art 55', 'art 53', 'art 52']

Patent US6686844 - Human interface system using a plurality of sensors - Google PatentsSearch Images Maps Play YouTube News Gmail Drive More »Sign inPatentsAn image sensor, a voice sensor, an auxiliary sensor part (infrared sensor, etc.), a total analyzing part, and an application communicate with each other through data/control signal communication units. Each sensor provides feedback on its signal detection results and control information used by the...http://www.google.com/patents/US6686844?utm_source=gb-gplus-sharePatent US6686844 - Human interface system using a plurality of sensorsAdvanced Patent SearchPublication numberUS6686844 B2Publication typeGrantApplication numberUS 10/337,818Publication dateFeb 3, 2004Filing dateJan 8, 2003Priority dateFeb 23, 2001Fee statusPaidAlso published asUS6583723, US20020118232, US20030097190Publication number10337818, 337818, US 6686844 B2, US 6686844B2, US-B2-6686844, US6686844 B2, US6686844B2InventorsMasaki Watanabe, Kentaro Murase, Takuya Noda, Kazuhiro WatanabeOriginal AssigneeFujitsu LimitedExport CitationBiBTeX, EndNote, RefManPatent Citations (5), Referenced by (68), Classifications (13), Legal Events (4) External Links: USPTO, USPTO Assignment, EspacenetHuman interface system using a plurality of sensors
US 6686844 B2Abstract
What is claimed is: 1. A human interface processing method using a plurality of sensors, comprising:
at least two kinds of sensor processings, each determining a range of a detection target and a detection sensitivity and acquiring a particular detection signal from the detection target at the detection sensitivity, the respective detection signals acquired by the sensor processings being of different types; total analyzing processing, investigating whether or not there is inconsistency among signal detection results detected by the respective sensor processings, and generating control information for the respective sensor processings; application processing utilizing the signal detection results obtained by the respective sensor processings; and communication processing communicating data and control information between the respective sensor processings, between the respective sensor processings and the total analyzing processing, and between the total analyzing processing and the application processing, wherein each of the sensor processings uses either of, or a combination of: the signal detection results or control information obtained from the other sensor processings, and the control information obtained from the total analyzing processing, thereby determining a range of a detection target and a detection sensitivity at a time of subsequent signal acquisition; each of the sensor processings outputs its signal detection results, and control information used by the other sensor processings to determine a range of a detection target and a detection sensitivity at a time of subsequent signal acquisition to the other sensor processings through the communication processing, and the total analyzing processing outputs control information used by each of the sensor processings to determine a range of a detection target and a detection sensitivity at a time of subsequent signal acquisition through the communication processings. 2. A human interface processing method using a plurality of sensors according to claim 1, wherein the detection target is a human being, and the sensor processings include at least image sensor processing and voice sensor processing,
a detection signal of the image sensor processing is human recognition information, the image sensor processing includes action recognition processing for interpreting an action of the detection target based on picture recognition results and recognizing a command inputted through a gesture, a detection signal of the voice sensor processing is human voice recognition information, and the voice sensor processing includes voice recognition processing for interpreting a voice of the detection target based on voice recognition results and recognizing a command inputted through a voice. 3. A human interface processing method using a plurality of sensors according to claim 1, wherein:
the detection target is a human being; the sensor processings include at least image sensor processing and auxiliary sensor processing; a detection signal of the image sensor processing is human picture recognition information; the image sensor processing includes action recognition processing interpreting an action of the detection target based on picture recognition results and recognizing a command inputted through a gesture; and a detection signal of the auxiliary sensor processing is information useful for detecting human position information. 4. A human interface processing method using a plurality of sensors according 3, wherein one of, or a combination of, ultrasonic sensor processing, infrared sensor processing, and range sensor processing is used as the auxiliary sensor processing.
5. A human interface processing method using a plurality of sensors according to claim 1, wherein the detection target is a human being, and:
the sensor processings include at least voice sensor processing and auxiliary sensor processing; a detection signal of the voice sensor processing inhuman voice recognition information; the voice sensor processing includes voice recognition processing for interpreting a voice of the detection target based on voice recognition results and recognizing a command inputted through a voice; and a detection signal of the auxiliary sensor processing is information useful for detecting human position information. 6. A human interface processing method using a plurality of sensors according 5, wherein either of, or a combination of, ultrasonic sensor processing, infrared sensor processing, and range sensor processing is used as the auxiliary sensor processing.
7. A human interface processing method using a plurality of sensors according to claim 1, wherein the detection target is a human being, and:
the sensor processings include at least image sensor processing, voice sensor processing, and auxiliary sensor processing; a detection signal of the image sensor processing is human picture recognition information; the image sensor processing includes action recognition processing interpreting an action of the detection target based on picture recognition results and recognizing a command inputted through a gesture; and a detection signal of the voice sensor processing is human voice recognition information; the voice sensor processing includes voice recognition processing interpreting a voice of the detection target based on voice recognition results and recognizing a command input by a voice input; and a detection signal of the auxiliary sensor processing is information useful for detecting human position information. 8. A human interface processing method using a plurality of sensors according 7, wherein one of, or a combination of, ultrasonic sensor processing, infrared sensor processing, and range sensor processing is used as the auxiliary sensor processing.
This application is a continuation of application Ser. No. 09/940,661, filed Aug. 29, 2001 U.S. Pat. No. 6,583,723, now allowed.
The conventional technique described in JP 2000-32674 A uses capture information from a camera that corresponds to an eye and voice information from a microphone that corresponds to an ear of an apparatus or a robot; however, they are used independently. A block diagram of FIG. 10 disclosed in JP 2000-32674 A does not show that information is exchanged between picture information processing and voice-information processing. Therefore, the technique described in JP 2000-32674 A has a problem that a picture of a person or a mannequin may be recognized as a human being, and voice information from a loudspeaker of acoustic equipment may be recognized as a human voice. Such matters are not intended by man-machine interface. A picture of a person, a mannequin, and a sound other than a human voice may become a noise for picture recognition and voice recognition, which decreases a recognition ratio. Furthermore, undesired information processing is conducted for inputting picture information and voice information obtained from an undesired target, which decreases a processing speed.
According to the second sensor signal acquisition control and signal analysis, weight is further laid on the feedback of the analysis results and the control instruction among the other sensors and the total analyzing part 50, compared with the first sensor signal acquisition control and signal analysis. In the sensor signal acquisition control, each sensor obtains an instruction on a range of a target and a sensitivity for signal acquisition in an environment from the other sensors or the total analyzing part 50, through the data/control signal communication units 70 to 75, and controls its sensor apparatus in accordance with the instruction. In the case where the sensor has not obtained such an instruction at the beginning of control, if required, the sensor in question obtains the analysis results of the other sensors, the results of the total analyzing part 50, and the past analysis results of the sensor in question, through the data/control signal communication units 70 to 75, and determines a range of a target and a sensitivity for signal acquisition in an environment by referring to those results, thereby controlling its sensor apparatus. In the signal analysis, if required, the analysis results of the other sensors, the results of the total analyzing part 50, and the past analysis results of the sensor in question are obtained through the data/control signal communication units 70 to 75, and a signal is analyzed by referring to those results. Furthermore, based on the analysis results,/if required, instruction data on a range of a target and a sensitivity for signal acquisition in an environment is given to the other sensors through the data/control signal communication units 70 to 75.
A human interface system using a plurality of sensors of Embodiment 2 according to the present invention is obtained by modifying the configuration in Embodiment 1. In the configuration shown in FIG. 1 described in Embodiment 1, there are provided the image sensor 20, the voice sensor 30, and the auxiliary sensor part 40. FIGS. 5A to 5C show variations of the configuration in Embodiment 1. FIG. 5A shows the configuration shown in FIG. 1 with the auxiliary sensor part 40 omitted. FIG. 5B shows the configuration shown in FIG. 1 with the voice sensor 30 omitted. FIG. 5C shows the configuration shown in FIG. 1 with the image sensor 20 omitted.
The voice recognizing part 333 analyzes a voice input detected by the sound source detecting part 31, and recognizes a command inputted through a voice. In recognizing a voice, the voice pattern storage part 334 matches a voice with data, and recognizes a command inputted by the person through a voice.
Hereinafter, person's position detection processing in the auxiliary sensor part 40 a will be described when the auxiliary sensor part 40 a receives data and feedback of a control instruction from the image sensor 20 a, the voice sensor 30 a, and the total analyzing part 50 a. Detection of a person's position based on picture information and voice information is conducted using object position information having a person's video detected by the image sensor 20 a, sound source position information detected by/the voice sensor 30 a, and a signal of the person detected by the auxiliary sensor part 40 a at a predetermined time.
Next, an example in which the auxiliary sensor part 40 a utilizes information from the capture apparatus and the recording apparatus will be described. In the case where the capture apparatus 21 and the recording apparatus 31 detects the position of a person and the position of a sound source by the function of each apparatus irrespective of the positional information from an ultrasonic sensor, the ultrasonic sensor scans the vicinity of the position based on the positional information and detects the presence absence of a moving object (person). Without an auxiliary sensor, the picture apparatus may recognize a portion other than a human being as the position of a human being, and the recording apparatus may detect a sound from a non-human sound source such as a TV and audio equipment. The ultrasonic sensor has an effect of preventing such erroneous detection.
The contents of the command understood as described above are provided to the application 60 a. The information on the position, direction, and posture of a person recognized by the person's status recognizing part 53, the information representing a particular person in the person identifying part 52, and the information on the command understood in the command understanding part 55 are given to the application 60 a, in the case where there is an inquiry from the application 60 a. Alternatively, instead of waiting for an inquiry from the application 60 a, as soon as the information is obtained, it may be sent to the application 60 a via the communication part 57. Alternatively, when the required information is set, it may be sent to the application 60 a. Next, generation processing of a command dictionary in the total analyzing part 50 a will be described. Prior to the beginning of the application 60 a, a link representing the relation between each command to be accepted by the application 60 a and commands in the action pattern storage part 234 and the voice pattern storage part 334 is stored in the command dictionary storage part 54. Furthermore, a list of commands that can be accepted by the application 60 a in the command dictionary storage part 54 is set by the application 60 in accordance with the situation, after the application 60 a is started. By nullifying the list, commands can be prevented from being accepted.
In the exemplary configuration of Embodiment 4 shown in FIG. 8, an action command, a voice command, and a person's status of an unspecified person. Compared with the exemplary configuration shown in FIG. 6 described in Embodiment 3, the configuration of the total analyzing part 50 b is different. In the configuration of Embodiment 4, the total analyzing part 50 b does not include a person's information storage part 51 and a person identifying part 52. Furthermore, the command understanding part 55 receives the results of the person's status recognizing part 53 directly, not via the person identifying part 52.
Patent CitationsCited PatentFiling datePublication dateApplicantTitleUS4961177Jan 27, 1989Oct 2, 1990Kabushiki Kaisha ToshibaMethod and apparatus for inputting a voice through a microphoneJP2000326274A Title not availableJPH1031551A Title not availableJPH1141577A Title not availableJPH01195499A Title not availableReferenced byCiting PatentFiling datePublication dateApplicantTitleUS7417551 *Mar 13, 2006Aug 26, 2008Omron CorporationPhotographic subject authenticating device, portable telephone, photographic subject authenticating method and photographic subject authenticating programUS7664571Feb 16, 2010Honda Motor Co., Ltd.Controlling a robot using poseUS7702624Apr 19, 2005Apr 20, 2010Exbiblio, B.V.Processing techniques for visual capture data from a rendered documentUS7706611Aug 23, 2005Apr 27, 2010Exbiblio B.V.Method and system for character recognitionUS7707039Dec 3, 2004Apr 27, 2010Exbiblio B.V.Automatic modification of web pagesUS7742953Jun 22, 2010Exbiblio B.V.Adding information or functionality to a rendered document via association with an electronic counterpartUS7812860Oct 12, 2010Exbiblio B.V.Handheld device for capturing text from both a document printed on paper and a document displayed on a dynamic display deviceUS7818215May 17, 2005Oct 19, 2010Exbiblio, B.V.Processing techniques for text capture from a rendered documentUS7831912Nov 9, 2010Exbiblio B. V.Publishing techniques for adding value to a rendered documentUS7990556Feb 28, 2006Aug 2, 2011Google Inc.Association of a portable scanner with input/output and storage devicesUS8005720Aug 23, 2011Google Inc.Applying scanned information to identify contentUS8019648Sep 13, 2011Google Inc.Search engines and systems with handheld document data capture devicesUS8081849Feb 6, 2007Dec 20, 2011Google Inc.Portable scanning and memory deviceUS8179563Sep 29, 2010May 15, 2012Google Inc.Portable scanning deviceUS8214387Jul 3, 2012Google Inc.Document enhancement system and methodUS8261094Aug 19, 2010Sep 4, 2012Google Inc.Secure data gathering from rendered documentsUS8346620Jan 1, 2013Google Inc.Automatic modification of web pagesUS8418055Apr 9, 2013Google Inc.Identifying a document by performing spectral analysis on the contents of the documentUS8442331Aug 18, 2009May 14, 2013Google Inc.Capturing text from rendered documents using supplemental informationUS8447066Mar 12, 2010May 21, 2013Google Inc.Performing actions based on capturing information from rendered documents, such as documents under copyrightUS8489624Jan 29, 2010Jul 16, 2013Google, Inc.Processing techniques for text capture from a rendered documentUS8505090Feb 20, 2012Aug 6, 2013Google Inc.Archive of text captures from rendered documentsUS8515816Apr 1, 2005Aug 20, 2013Google Inc.Aggregate analysis of text captures performed by multiple users from rendered documentsUS8570330 *Sep 22, 2011Oct 29, 2013Sony CorporationImage processing device, image processing method and information storage mediumUS8600196Jul 6, 2010Dec 3, 2013Google Inc.Optical scanners, such as hand-held optical scannersUS8620083Oct 5, 2011Dec 31, 2013Google Inc.Method and system for character recognitionUS8638363Feb 18, 2010Jan 28, 2014Google Inc.Automatically capturing information, such as capturing information using a document-aware deviceUS8713418Apr 12, 2005Apr 29, 2014Google Inc.Adding value to a rendered documentUS8781228Sep 13, 2012Jul 15, 2014Google Inc.Triggering actions in response to optically or acoustically capturing keywords from a rendered documentUS8799099Sep 13, 2012Aug 5, 2014Google Inc.Processing techniques for text capture from a rendered documentUS8831365Mar 11, 2013Sep 9, 2014Google Inc.Capturing text from rendered documents using supplement informationUS8831954 *May 5, 2009Sep 9, 2014Nuance Communications, Inc.Localizing the position of a source of a voice signalUS8874504Mar 22, 2010Oct 28, 2014Google Inc.Processing techniques for visual capture data from a rendered documentUS8892495Jan 8, 2013Nov 18, 2014Blanding Hovenweep, LlcAdaptive pattern recognition based controller apparatus and method and human-interface thereforeUS8953886Aug 8, 2013Feb 10, 2015Google Inc.Method and system for character recognitionUS8990235Mar 12, 2010Mar 24, 2015Google Inc.Automatically providing content associated with captured information, such as information captured in real-timeUS9008447Apr 1, 2005Apr 14, 2015Google Inc.Method and system for character recognitionUS9030699Aug 13, 2013May 12, 2015Google Inc.Association of a portable scanner with input/output and storage devicesUS9075779Apr 22, 2013Jul 7, 2015Google Inc.Performing actions based on capturing information from rendered documents, such as documents under copyrightUS9081799Dec 6, 2010Jul 14, 2015Google Inc.Using gestalt information to identify locations in printed informationUS9116890Jun 11, 2014Aug 25, 2015Google Inc.Triggering actions in response to optically or acoustically capturing keywords from a rendered documentUS9143638Apr 29, 2013Sep 22, 2015Google Inc.Data capture from rendered documents using handheld deviceUS9268852Sep 13, 2012Feb 23, 2016Google Inc.Search engines and systems with handheld document data capture devicesUS9275051Nov 7, 2012Mar 1, 2016Google Inc.Automatic modification of web pagesUS9323784Dec 9, 2010Apr 26, 2016Google Inc.Image search using text-based elements within the contents of imagesUS20060026078 *Apr 1, 2005Feb 2, 2006King Martin TCapturing text from rendered documents using supplemental informationUS20060029296 *Apr 1, 2005Feb 9, 2006King Martin TData capture from rendered documents using handheld deviceUS20060041484 *Apr 1, 2005Feb 23, 2006King Martin TMethods and systems for initiating application processes by data capture from rendered documentsUS20060041538 *Apr 1, 2005Feb 23, 2006King Martin TEstablishing an interactive environment for rendered documentsUS20060041605 *Apr 1, 2005Feb 23, 2006King Martin TDetermining actions involving captured information and electronic content associated with rendered documentsUS20060041828 *Apr 1, 2005Feb 23, 2006King Martin TTriggering actions in response to optically or acoustically capturing keywords from a rendered documentUS20060047639 *Apr 1, 2005Mar 2, 2006King Martin TAdding information or functionality to a rendered document via association with an electronic counterpartUS20060050996 *Apr 1, 2005Mar 9, 2006King Martin TArchive of text captures from rendered documentsUS20060053097 *Apr 1, 2005Mar 9, 2006King Martin TSearching and accessing documents on private networks for use with captures from rendered documentsUS20060081714 *Aug 23, 2005Apr 20, 2006King Martin TPortable scanning deviceUS20060098899 *Sep 27, 2005May 11, 2006King Martin THandheld device for capturing text from both a document printed on paper and a document displayed on a dynamic display deviceUS20060098900 *Sep 27, 2005May 11, 2006King Martin TSecure data gathering from rendered documentsUS20060122983 *Aug 18, 2005Jun 8, 2006King Martin TLocating electronic instances of documents based on rendered instances, document fragment digest generation, and digest based document fragment determinationUS20060208882 *Mar 13, 2006Sep 21, 2006Omron CorporationPhotographic subject authenticating device, portable telephone, photographic subject authenticating method and photographic subject authenticating programUS20060256371 *Feb 28, 2006Nov 16, 2006King Martin TAssociation of a portable scanner with input/output and storage devicesUS20060271239 *Apr 17, 2006Nov 30, 2006Gonzalez-Banos Hector HControlling a robot using poseUS20070279711 *Feb 6, 2007Dec 6, 2007King Martin TPortable scanning and memory deviceUS20070300142 *Jun 6, 2007Dec 27, 2007King Martin TContextual dynamic advertising based upon captured rendered textUS20080313172 *Jan 10, 2008Dec 18, 2008King Martin TDetermining actions involving captured information and electronic content associated with rendered documentsUS20100277579 *Apr 29, 2010Nov 4, 2010Samsung Electronics Co., Ltd.Apparatus and method for detecting voice based on motion informationUS20110033080 *Feb 10, 2011Exbiblio B.V.Processing techniques for text capture from a rendered documentUS20110054909 *May 5, 2009Mar 3, 2011Koninklijke Philips Electronics N.V.Localizing the position of a source of a voice signalUS20120007893 *Jan 12, 2012Sony Computer Entertainment Inc.Image processing device, image processing method and information storage medium* Cited by examinerClassifications U.S. Classification340/573.1, 340/686.1, 340/573.7, 340/5.1International ClassificationG06T7/20, G06F3/16, G10L15/00, G06T7/00, G06F3/01, G07C9/00Cooperative ClassificationG06F3/017, G07C9/00158European ClassificationG07C9/00C2DLegal EventsDateCodeEventDescriptionJun 22, 2004CCCertificate of correctionJul 6, 2007FPAYFee paymentYear of fee payment: 4Jul 6, 2011FPAYFee paymentYear of fee payment: 8Jul 22, 2015FPAYFee paymentYear of fee payment: 12RotateOriginal ImageGoogle Home - Sitemap - USPTO Bulk Downloads - Privacy Policy - Terms of Service - About Google Patents - Send FeedbackData provided by IFI CLAIMS Patent Services