Computer-based method and system for providing active and automatic personal assistance using an automobile or a portable electronic device

A method and a system for providing personal assistance in daily activities. A method and a system for automatically and actively providing personal assistance, using a vehicle or a portable electronic device, based on detected data regarding a user and the user's environment. The method and system may include a processor, at least one sensor, an output device, a communications unit, and a database. The database may further include a memory and cloud-based database and computing. The method and system may actively and automatically provide personal assistance regarding health, exercise, diet, or nutrition. The method and system may assist the user or a health professional in health diagnosis and treatment.

BACKGROUND

The present invention relates generally to a computer-based method of providing personal assistance in daily activities and more particularly, pertains to a method and a system of automatically and actively providing personal assistance, using a vehicle or a portable electronic device, based on detected data regarding the user and the user's environment.

2. Description of the Related Art

Computing power is ever advancing. Records of data may be made with more ease than was previously obtainable. Prior personal assistance and memory jogging methods/systems implemented in a vehicle or a portable electronic device have heavily relied upon passively outputting data based on an input/request from the user or upon occurrence of a predetermined or scheduled event stored in a database. However, currently, a computer-based method or system is needed for actively and automatically providing personal assistance to the user (e.g., actively and automatically reminding the user of an event or actively and automatically encouraging the user to take a particular action based on conversational and situational cues). Thus, a method/system is needed for detecting and analyzing data regarding the user and the environment surrounding the user, and actively and automatically providing personal assistance based on the detected and analyzed data.

SUMMARY

The present invention relates generally to a computer-based method of providing personal assistance in daily activities and more particularly, pertains to a method and a system of automatically and actively providing personal assistance, using a vehicle or a portable electronic device, based on detected data regarding the user and the user's environment.

In one embodiment, the present invention may be, for example, a computer-based method including the steps of: detecting, using at least one sensor, a data associated with at least one person, an object, a living being, a place, an environment, or combinations thereof within a proximity of at least one of the at least one sensor; selectively storing, using a processor, a learned data in a database based on the detected data and a pre-programmed algorithm stored in the database; passively outputting, using a communications unit or an output device coupled to or in communication with the processor, a first output data upon a request received by the processor or a predetermined or scheduled event stored in the database; and actively and automatically outputting, using the communications unit or the output device, a second output data based on the detected data, the learned data, and the pre-programmed algorithm.

In another embodiment, the method may include analyzing, using the processor, the detected data based on the learned data corresponding to the at least one person, the object, the living being, the event, the place, the environment, or combinations thereof such that the steps of selectively storing the database and actively and automatically outputting the second output data are further based on the analyzed data. In yet another embodiment, the method may include analyzing, using the processor, a conversation or an interaction based on the detected data, such that the step of actively and automatically outputting the second output data further includes at least one of: actively and automatically generating, using the communications unit or the output device, a sound, an image, a video, or combinations thereof for actively and automatically initiating or participating in the conversation or the interaction with the at least one person; or actively and automatically generating, using the communications unit or the output device, the sound, the image, the video, or combinations thereof for actively communicating a reminder to the user.

In yet another embodiment, the present invention may be a vehicle including: at least one sensor configured to detect a data associated with a driver, a passenger, an object, a living being, an event, a place, an environment, or combinations thereof within a proximity of at least one of the at least one sensor; a database storing a pre-programmed algorithm; a communications unit; and a processor coupled to or in communication with the at least one sensor, the database, and the communications unit, the processor being configured to: selectively store a learned data in the database based on the detected data and the pre-programmed algorithm; analyze the detected data based on the learned data and the pre-programmed algorithm; passively outputting, using the communications unit, a first output data upon a request received by the processor or a predetermined or scheduled event stored in the database; and actively and automatically output, using the communications unit, a second output data based on the analyzed data, the learned data, and the pre-programmed algorithm.

In yet another embodiment, the present invention may be a portable electronic device including: at least one sensor configured to detect a data associated with at least one person or a user, an object, a living being, an event, a place, an environment, or combinations thereof within a proximity of at least one of the at least one sensor; a database storing a pre-programmed algorithm; a communications unit; and a processor coupled to or in communication with the at least one sensor, the database, and the communications unit, the processor being configured to: selectively store a learned data in the database based on the detected data and the pre-programmed algorithm; analyze the detected data based on the learned data and the pre-programmed algorithm; passively output, using the communications unit, a first output data upon a request received by the processor or a predetermined or scheduled event stored in the database; and actively and automatically output, using the communications unit, a second output data based on the analyzed data, the learned data, and the pre-programmed algorithm.

DETAILED DESCRIPTION

Referring toFIG. 1, a block diagram is shown of a method/system100according to an embodiment of the present invention. The method/system100may include a processor102, a communications unit104, sensors106, a database112, and an output device114. The database112may include a memory108and cloud-based database and computing110. In other embodiments, the database112may include solely the memory108or solely the cloud-based database and computing110. The various units of the method/system100may be in communication with one another by using transmission of an electronic signal through a Control Area Network (CAN) bus. In other embodiments, the control and communications may be over various other types of serial communication links, direct wirings, digital communication buses, wireless communications, or other communication links and networks.

In general, in the method/system100, the database112stores a pre-programmed algorithm. The method/system100continuously detects a data associated with a user or a person within a proximity of at least one of the sensors106. The method/system100may further detect data associated with a static or dynamic object, a living being, a surrounding environment/place, and other detectable matters within the proximity of at least one of the sensors106. The method/system100not only outputs a first output data (using the communications unit104and/or using the output device114) upon a request received by the processor102or a predetermined event stored in the database, but also actively and automatically outputs a second output data (via the communications unit104and/or using the output device114). As such, the method/system100is configured to actively and automatically provide personal assistance to the user by detecting data regarding the user and the surrounding environment of the user, analyzing, storing, and cataloging the data, and outputting a second output data which can, for example, include helpful information as needed and as appropriate under the circumstances as determined by appropriateness algorithm rules described below (including but not limited to privacy concern rules). The processor102may modify or supplement the appropriateness algorithm rules based on the learned data. Analyzing and variants thereof, as used herein, may refer to including, but not limited to, any analysis, parsing, cataloging, or any other type of processing performed using detected data that may assist in storing data for later use, analyzing data to draw an inference, retrieving data, or outputting data.

The method/system100is not limited to any particular device, unit, or platform. In another embodiment, the method/system100is integrated in a vehicle. A vehicle as used herein includes, but is not limited to, any type of transportation device such as an automobile, a wheelchair, a walker, a two-wheeled self-balancing vehicle, a scooter, a three-wheeled vehicle, or various other devices which may enable or assist a user in transportation.

In an embodiment illustrated inFIGS. 2A and 2B, a computer-based method/system implemented in a vehicle for detecting and analyzing conversations and events, and actively and automatically providing assistance and guidance to the user, according to another embodiment of the present invention. The processor102may be an Electronic Control Unit (ECU) of the vehicle or any other processor capable of processing data. The memory108may be employed in the vehicle and coupled to or in communication with the processor102. The memory108may be, for example, a random access memory (RAM) or a read only memory (ROM). The ECU or a separate processor may have access to the Internet or be in communication with the cloud-based database and computing110. Various cloud-based solutions may be utilized without limiting the scope of the invention. The communications unit104or the output device114may be an input/output interface210illustrated inFIG. 2A. The sensors106may be located at sensor locations220in order to detect data.

Referring toFIG. 1, in yet another embodiment, the method/system100may be a software program or application (“app”) implemented in a portable electronic device. A portable electronic device as used herein includes, but is not limited to, a laptop, a mobile communication device such as a cellular phone or a smart phone, a tablet, a handheld PC, a PDA, a smartbook, and other portable electrical or electromechanical devices capable of processing data.

Referring toFIG. 1, in one embodiment, method/system100may utilize the sensors106to detect data within a proximity of the user, thereby allowing the processor102to draw inferences regarding the surrounding environment of the user. The processor102may then analyze the detected data in order to extract information and draw inferences that may enable the method/system100to assist the user in real time or store the analyzed data for automatically providing personal assistance at a later time. Such detected data may be related to persons, objects, events, places or any other information that the sensors106may detect in order to enable the processor102to draw inferences regarding any matter in an environment within a proximity of at least one of the sensors106.

The sensors106may include converters that measure a physical quantity and convert the measured physical quantity into a signal which can be analyzed or processed by the processor102. The sensors106may be any type of sensor and/or a plurality of sensors operating in concert together. The sensors106may be integrated in or coupled to the method/system100. The sensors106may be positioned at a distance away from the method/system100such that the sensors106are in communication with the processor102, e.g., via a network. The processor102may be coupled to or in communication with external devices in order to supplement the data detected by the sensors106. The sensors106may be integrated in or coupled to an input/output communications interface, the output device114, or various other devices or platforms without limiting the scope of the invention. For example, the input/output communications interface may be integrated in or coupled to a vehicle or a portable electronic device.

Referring toFIGS. 2A and 2B, in the method/system200implemented in a vehicle, the sensors106may be positioned anywhere in the interior or the exterior of the vehicle without limiting the scope of the invention. For example, the sensors106may be positioned at sensor locations220shown inFIG. 2Aand/or sensor locations230shown inFIG. 2B. The sensors106may alternatively or in addition, be implemented in an external device in communication with or coupled to the vehicle.

Similarly, in an embodiment in which the method/system100is implemented in a portable electronic device, the sensors106may be integrated in the portable electronic device. The sensors106may alternatively or in addition, be implemented in an external device in communication with or coupled to the portable electronic device.

The detected data, as used herein, includes, but is not limited to, any detected data that may assist the processor102in drawing an inference regarding a user or a person. The detected data includes, but is not limited to, a physical appearance, motion, interaction, action, behavior, conversation, characteristic, location, identity, or combinations thereof of at least one person or a user within a proximity of the user or at least one of the plurality of the sensors106. The data, as used herein, may also include any data that may assist the processor102to draw an inference regarding any matter in an environment within a proximity of the user or at least one of the plurality of sensors106. The data includes, but is not limited to, a characteristic, location, condition, type, or any attribute associated with an object, living being, surrounding place/environment, and other matters therein within a proximity of at least one of the sensors106.

Proximity as used hereinafter refers to an area around the user or within the detection range of at least one of the sensors106within which data is sought to be collected. However, proximity, as used herein, is not limited to any particular distance range as the distance may vary depending on detection range of the sensors106, the extent of area in which data is sought to be collected, and various other design or practical factors or limitations. In another embodiment, the detected data may not be limited to an environment within a proximity of the user. For example, in one embodiment, GPS data may be detected regarding a location that may be positioned well beyond the detection range of all of the sensors106. The processor102may analyze the detected GPS data and automatically direct the communications unit104to output a second output data based on the detected GPS data.

In one embodiment, the sensors106may include sound and speech sensors such as a microphone capable of detecting speech, conversation, or sound. Speech as used herein may refer to speech, voice, or any other sound conveying information. In yet another embodiment, a 3-D (three-dimensional) microphone may be utilized to detect speech and further pinpoint or detect the location of the source of the detected speech.

For example, referring toFIG. 2A, a microphone may be positioned anywhere in the passenger compartment of the vehicle without limiting the scope of the invention. For example, the microphone may be positioned at any of the sensor locations220. The microphone may be positioned, for example, on the steering wheel or an area adjacent to the steering wheel. The microphone or an additional microphone may be positioned, for example, in the driver or passenger overhead. The microphone may be utilized to detect a speech of the user202and/or the person206. The detected speech may be analyzed as described in details below with respect toFIGS. 3-7and9.

The sensors106may further include an image sensor or a camera that may be integrated in, coupled to, or in communication with the method/system100for capturing images/videos. In an embodiment implemented in a portable electronic device, a camera embedded in the portable electronic device may be utilized to capture an image or a video. For example, if the method/system100is implemented in a smartphone, mobile applications may be operated to add capabilities to the camera or to process the image or video captured by the camera.

A camera embedded in a portable electronic device may have limited vantage points or a blocked view at times. In one embodiment, when the user is indoors, a camera may be utilized as an external device placed in a stationary position that has a better vantage point for capturing and sensing data. The camera may be in communication with the processor102, thereby transmitting the detected data to the processor102. For example, the camera may supplement the detected data regarding the at least one person or user by capturing a visual facial feature of the user/person.

Referring toFIGS. 2A and 2B, the camera may be positioned, for example, at the sensor locations220. For example, the camera may capture an image or a video of the user202and/or the person206. In addition or alternatively, a camera may be positioned on the exterior of the vehicle at sensor locations230for detecting data regarding an object, a living being, an event, a place, an environment (e.g., data regarding steep grade or vibration), or combinations thereof within a proximity of the vehicle. During the normal course of transit of the vehicle, data may be detected using the sensors106regarding the user202or the person206and stored in the database112for automatically providing personal assistance in real time or at a later time.

The sensors106may include positional encoders, compasses, navigational, and GPS sensors. The method/system100may be in communication with or coupled to a GPS system for supplementing the detected data. For example, if the method/system100is mounted on a mobile unit (e.g., a vehicle/transportation device), the sensors106may include an inertial measurement unit (IMU), which detects velocity, orientation, and gravitational forces of the mobile unit, using a combination of accelerometers, compasses, distance sensors, geomagnetic sensors, and gyroscopes. The sensors106may include various proximity/position sensors.

The sensors106may further include a capacitive displacement sensor, a passive thermal infrared sensor, a photocell (reflective) sensor, a radar sensor, a vibration sensor, a sonar sensor, and/or the like. The sensors106may also be configured to provide a user's current location and identify objects in the vicinity and/or within a geographic range. The sensors106may include perimeter monitoring sensors or ultraviolet, radio frequency, or infrared sensors. A three dimensional (3-D) scanner may be employed to analyze a real-world object or environment to collect data on its shape and its appearance (e.g., color). The detected data is then communicated to the processor102for constructing digital 3-D models.

The sensors106may further detect atmospheric pressure, smoke, and various other attributes of the environment within a proximity of the user. The sensors106may utilize thermistors to measure temperature. Based on the detected data, the processor102may determine the type of the environment (e.g., a shopping mall, a parking lot, an office, and other place/environment characteristics). For example, the processor102may further determine the current season or weather and the current temperature or moisture content based on the detected data. The sensors106may further include tactile sensors utilized to analyze contact with an object, person, living being, and other matters therein. In one embodiment, a touch sensor and/or a force sensor may be utilized to supplement the tactile information.

The sensors106may include sensors capable of detecting odor, magnetic fields, or radiation. Various other sensors may be integrated in, coupled to, or in communication with the sensors106in order to improve the quality or increase the quantity of the detected data without limiting the scope of the present invention.

Prior to a further detailed description of various embodiments of the method/system100, an example of an application of the method/system100is provided.

Referring toFIG. 3, the method/system300serves as a personal assistant in daily activities. When the method/system300recognizes that the user302is searching for an object, i.e. a key314, the portable electronic device304directs the user302to the key314by looking up prior relevant learned data and inferring context and interaction points. In an embodiment, the method/system300recognizes that the user302is searching for the key314and draws an inference310that the key314was placed on the table312, by looking up prior relevant learned data. The method/system300directs the output device114(e.g., a speaker) to output a second output data corresponding to relevant helpful information regarding the location of the key314. The output device114may be integral to, coupled to, or in communication with the method/system300.

Outputting relevant helpful information includes outputting a first output data upon an input/request from the user or upon occurrence of a predetermined/scheduled event, and actively and automatically outputting a second output data based on the detected data and further based on a learned data. The learned data as used herein refers to currently or previously detected, processed, and stored data. The processor102may direct the portable electronic device304to passively output a first output data when the processor102receives, using the sensors106, or determines via the detected data a request by the user302inquiring about the location of the key314. The input/request may be received via: recognition of a facial expression detected by a camera and analyzed by the processor102; input/request received from the user302using a communication interface of the portable electronic device304; an input/request from external devices coupled to or in communication with the portable electronic device304; other devices and methods of detecting an input/request; or combinations thereof.

However, an important aspect of the invention is that the method/system300is further configured to automatically output a second output data even without an input/request received from the user302and without occurrence of a predetermined or scheduled event. Recognition of the fact that the user302is searching for the key314can be performed using a combination of the sensors106described above. The method/system300is not limited to a particular sensor of the sensors106or any particular means of detecting such data.

In one embodiment, for example, if the user302utters a speech/thought306, the sensors106detect the speech/thought306. The processor102of the portable electronic device304can analyze the speech/thought306by parsing the elements of the speech/thought306to determine that the user302is attempting to search for the key314. Additionally or alternatively, a camera may be integrated in or coupled to the portable electronic device304. Alternatively or in addition, the portable electronic device304may detect a facial expression of the user302indicating a sense of confusion as to the current location of the key314. Other sensed data such as tactile data may be detected as well. By looking up prior relevant learned data about the key314, the method/system300matches speech elements with the object (key314) and actions, place, time, and the like associated with the object (key314). The processor102of the method/system300directs the portable electronic device304to output a second output data corresponding to relevant and appropriate information for assisting the user302in finding the key314.

For example, the portable electronic device304may generate an output reminder/speech308indicating to the user302that the key314is placed on the table312. The portable electronic device304may, alternatively or in addition, display an output video recorded (or recording) of the key314or the action of the user302placing the key314on the table312. In one embodiment, the user302is only informed of the location of the object (e.g., the key314).

For example, using the sensors106, data is continuously or intermittently detected regarding a user302or a person during daily activities. The sensors106further continuously or intermittently detect data regarding static or dynamic objects, living beings, place/environment, or combinations thereof within a proximity of the sensors106. During daily activities of the user302, the method/system300may collect data associated with actions, interaction, speech, conversation, location, and other characteristics of the user302and the surrounding environment within a proximity of the user302.

For example, the method/system300may have recognized, using the sensors106and the processor102, that the key314was being placed on the table312based on a pre-programmed algorithm and previously detected, processed, and stored data as discussed in detail below. Given the detected data regarding a person, the user302, objects (e.g., the key314or the table312), places (e.g., the home environment around the user302), the speech/thought306, other detected or stored data, or combinations thereof, the processor102of the method/system300can infer context and key interaction points or events. For example, a key interaction point in the embodiment shown inFIG. 3, may have been the event of “<person> placing <object1> on <object2>.” The <person>, <object1>, and <object2> may correspond to the user302, the key314, and the table312, respectively. The event of “<person> placing <object1> on <object2>” may be inferred from a sequence of detected events as follows: “<person> holding and/or touching <object1>,” “<object1> touching and positioned above <object2>”, “<person> at a distance from or not touching <object1>.” Similarly, picking up an object may be recognized. For example, the event “<person> picking up <object1>” can be composed of a sequence such as “<object1> next to <person>,” “<person> touches <object1>,” and “<person> holding <object1>.” As such, the method/system300may utilize algorithms of the like to automatically detect and analyze relatively high level events by recognizing a sequence of relatively lower level events. The algorithms may be adjusted or developed as the method/system300collects more data regarding the user302and the matters within the surrounding environment of the user302.

The recognition of placement of the key314on the table312may have been via analyzing a conversation or speech indicating placement of the key314on the table312. The recognition may alternatively or in addition be based on capturing image/video of the placement of the key314. Upon using such a process for recognizing the various events within the surrounding environment of the user302, the method/system300can create a journal summary of daily life of the user302.

The method/system300may be applied in various applications and with respect to various objects and events to provide personal assistance to the user302. For example, in an alternative embodiment, the method/system300may determine based on the combinations of data included in the detected data whether the user302is thirsty or forgetful of where a bottle of water has been placed. The processor102of the portable electronic device304can direct the communications unit104to output a second output data containing information as to the location of the water bottle.

Referring to a flowchart of a method/system400shown inFIG. 4, in step404, data is detected, using the sensors106as described above with respect toFIGS. 1-3. The method/system400is only an example of the detection and analysis of the data. The steps of the method/system400may be performed simultaneously or in various combinations of orders. The degree to which the data can be collected regarding the surrounding environment of the user and matters therein may depend on what or which sensors106are available and the processing limitations of the processor102and/or the cloud-based database and computing110. As such, the method/system400may be adjusted accordingly in real time by monitoring such limitations.

The processor102may work in concert with the sensors106for improving collection of the data. The processor102may also consider whether the user or a person is requesting or attempting to convey information. For example, if a user is making a facial expression without speech to communicate with the method/system400, the processor102can direct the communications unit104to pose follow-up questions or inquiries using the communications unit104in order to supplement or clarify the detected data. For example, the method/system400may direct an output speech to be generated, thereby asking the user to clarify the facial expression. The user may then respond in a voice command clarifying the conveyed information.

In step406, data regarding dynamic objects, living beings, and/or place/environment is detected and inferences are drawn accordingly. In one embodiment, if a camera is utilized in the method/system400and the camera has vantage points to detect images/videos containing useful information regarding the surrounding environment of the user, the method/system400may utilize the camera in addition to or as an alternative to a microphone to detect the data regarding the surrounding environment of the user and matters therein. For example, in step406, the method/system400may detect movement or changes in a scene or other dynamic regions as observed by cameras in order to focus the sensors106on the detected dynamic regions. The processor102classifies the detected dynamic region as described below.

For example, detection of a person, living being, and/or a dynamic object may be performed by looking for changes in data detected by the sensors106. For example, changes in data received from the camera may be determined. Changes in data detected by the sensors106may be identified by first estimating the motion of the method/system400or a mobile platform upon which the method/system400is implemented, using GPS, IMU, or techniques such as visual odometry which allow estimation of the motion of a camera by tracking corner or blob features between two camera frames. As such, the method/system400may identify motion in the surrounding environment of the user which does not correspond to the motion of the method/system400.

Upon identifying the changing parts of the scene within the detected data, the method/system400seeks to recognize the changing elements, using techniques such as “eigenfaces” and “skeletal recognition” to recognize persons and faces. Additionally, standard techniques like Support Vector Machines, Deformable Parts Model and dynamic programming can be used to learn different models for various object/person classes. The types of features that can be used in these recognition tasks can be any combination of features like SIFT (Scale-Invariant Feature Transform), SURF (Speeded Up Robust Features), Gist modeling, Sobel, Fast, and other features and/or techniques that enable the method/system400to recognize a person, object, living being, or place/environment within a proximity of the user.

Thus, by detecting the dynamic regions, a dynamic object or being can be detected in step406, and a new person entering the environment within the proximity of the user may be detected in step412and classified by the processor102in the database112accordingly. In step406, the method/system400may extract the remaining static regions of the detected image/video in order to gather data regarding static objects or a place/environment within a proximity of the user.

The sensors106may utilize a microphone and a speech recognition module to detect speech, conversation or interaction as shown in step408. The method/system400may further extract conversation elements containing useful data in step410. In step422, the processor102of the method/system400matches extracted conversation or speech elements from step410to the detected person, object, living being, place/environment, or combinations thereof.

In step428, the method/system400looks up prior relevant information based on context and based on the matched conversation events from step422regarding the person, object, living being, place/environment, or combinations thereof. In step426, the processor102stores relevant information for later use in the database112based on prior relevant information. For example, if the processor102detects facial features of a person entering the environment and also detects that the new person is speaking, the speech elements can be matched with the new person. Speech data related to the person may be stored in the database112for later use. A 3-D microphone or a microphone array may also be utilized to localize the origin of the sound or voice. The method/system400can track and log data related to the person in order to supplement the detected data. The method/system400may actively and automatically output a second output data in step430based on the matched conversation events to the corresponding person, object, living being, place/environment, or combinations thereof of step422and based on the inferred context and interaction key points from step424.

The processing of data (e.g., in steps406-430) can be performed by continuously analyzing data gathered by the sensors106in real time. The cloud-based database and computing110may be utilized due to restraints on the information storage capacity of the memory108or energy capacity challenges associated with processing using solely the processor102. However, in one embodiment, both on-board and off-board processing capabilities are utilized to prepare for events in which on-board processing may be preferable (e.g., a poor connection in cloud communications) to ensure a minimal level of capability. For example, if the method/system400is implemented in a portable electronic device that may not have sufficient capacity to perform the steps described herein, the cloud-based database and computing110can provide assistance in sharing the load of processing.

In step430, the processor102may passively output, using the communications unit104, a first output data upon an input/request received by the processor102or a predetermined or scheduled event stored in the database112.

The processor102may further actively and automatically output, using the communications unit104, a second output data based on the detected data, previously detected, stored, and processed data, the pre-programmed algorithm, or combinations thereof.

For example, the first or second output data may be displayed via a display screen or via generation of an output speech via a speaker. In an embodiment in which the method/system400is implemented in a portable electronic device, the input/output user interface, display, speakers, microphone, or combinations thereof may be utilized as the communications unit104. Referring toFIG. 2A, in an embodiment in which the method/system400is implemented in a vehicle, the input/output interface210, display, speakers, microphone, or combinations thereof of the vehicle may be utilized as the communications unit104.

The display may be, for example, a (color) liquid crystal display on which the second output data is displayed. The output images/videos may be displayed using an LCD, an organic light emitting display, a plasma display, light-emitting diodes, or any other display mechanism for displaying the output images/videos. In another embodiment, due to practical concerns, a projector in lieu of a display screen may be utilized to project output images/videos on a wall, screen, or surfaces of the like based on the first and/or second output data. A projector may be preferable if the method/system400is designed for a device with characteristics requiring a compact design.

The processor102may further submit the first and/or second output data to an output device114(e.g., another electronic device authorized by the user). In another embodiment, the output device114is coupled to or in communication with a device including actuators and capable of operating mechanical movements.

In an embodiment in which the method/system400is implemented in a vehicle as shown, for example,FIGS. 2A and 2B, a pre-programmed algorithm may be stored in the database112with respect to an output timing (e.g., when to generate an output speech and/or when to display an output image/video), a form (e.g., generating an output speech versus displaying an output video), and an information content of the outputted first and/or second output data. For example, the timing and information content of the actively and automatically generated output speeches or displayed output images/videos may be based on a determined expected level of driving attention desirable for driving the vehicle under the current circumstances. The determined expected level of driving attention may depend upon a current operation of the vehicle. For example, when the processor102determines that the vehicle is in motion, the processor102determines that the current expected level of driving attention has increased or is high. For example, the processor102may direct the communications unit104to generate an output speech when appropriate under the circumstances based on the detected data. On the contrary, when the processor102determines that the vehicle is not in motion and/or an engine of the vehicle is not operating, the processor102determines that the current expected level of driving attention has decreased. For example, under such circumstances, the processor102may direct the communications unit104to display a video in addition to or in lieu of generating the output speech.

Referring toFIG. 5, the method/system500, the portable electronic device504can utilize the sensors106(e.g., the microphone) to detect the data regarding the conversation between the user502and the person506. The processor102can parse the conversation and store relevant elements in the database112. An image or video of the person506may be captured using a camera and analyzed by the processor102as described above with respect to steps402-422ofFIG. 4. A first and/or second output data may be outputted using the process described above with respect to the method/system400.

Referring toFIG. 6A, in the method/system600, a portable electronic device604may provide personal assistance to the user602when visiting a health professional606for consultation, diagnosis, or treatment. A close-up view of the portable electronic device604is shown inFIG. 6Bin order to facilitate illustration of the display of the communications unit104for the portable electronic device604ofFIG. 6A.

As with other social encounters described herein, for example, if the portable electronic device604determines that the user602may not recall certain information with respect to the health professional606, the visitation, or other associated information, the portable electronic device604may output a second output data for informing the user602accordingly (if appropriate under the circumstances). For example, the portable electronic device604may output a second output data reminding the user602of the number of years that the user602and the health professional606have known each other and their previous encounters, if the portable electronic device604determines that outputting such information would be appropriate under the circumstances.

An important aspect of the method/system600is that the first and/or second output data may be based on privacy concerns of the user602(e.g., concerns of the user602regarding with whom, to what extent, and in what manner the user602would be comfortable in sharing such information under the current circumstances). Privacy concerns may be programmed in the form of a pre-programmed algorithm stored in the database112which may further be supplemented during operations of the method/system600based on the detected, processed, and stored data. Such privacy concerns may be based on combinations of previously detected and processed data or the pre-programmed algorithm for determining what type of information is appropriate to be outputted under the circumstances. In one embodiment, individuals may be distinguished in part by social interaction classes. For example, privacy concerns are less weighty when a conversation is with the health professional606, a family member or a therapist as discussed above. As such, some of the standards as to the extent of interaction with a trusted individual can be stored in pre-programmed data or algorithm, and some of the information regarding with whom and to what extent the method/system600should interact can be adjusted based on the detected data.

In one embodiment, the method/system600may associate a high level of trust with a person based on prior detected conversations of the user602with the person. That is, a detected friend having close/intimate relations with the user602can be distinguished from an acquaintance or a stranger based on prior conversations and interactions. Based in part on the recognized distinction, the portable electronic device604may adjust the degree and type of information included in the second output data. For example, if the processor102determines that because the user602would be comfortable with sharing certain information with the health professional606(acting in a professional capacity with expectations of upholding the privacy of the user602), the portable electronic device604would output a higher degree of or more private information than the portable electronic device604would if the interactions were directed to an individual encountered for the first time and not acting in a confidential and professional capacity. The method/system600can determine whether the detected person is a health professional606, therapist, or other professionals endowed with high level of trust and expectation of upholding privacy of communications based on the detected data, pre-programmed algorithm, and previously stored and processed data as discussed above with respect toFIG. 4.

In one embodiment as shown inFIG. 6, for example, the health professional606may inquire about the reason, the timing, or the cause of an arm injury sustained by the user602which has prompted visiting the health professional606by posing the speech/inquiry608to the user602. The speech/inquiry608may be “How and when did your arm get injured?” directed to the user602. The portable electronic device604may parse and analyze the conversation as described above in order to analyze the elements of the speech/inquiry608and the relationship, sequence, or correlation of the elements to one another based on previously detected, processed, and stored data. The portable electronic device604may search the database112regarding previous injuries sustained. The portable electronic device604may match the elements of “arm” and “injury” with a slip and fall accident leading to an arm injury which was sustained during a prior week and catalogued accordingly. After analyzing the elements of the speech/inquiry608, the portable electronic device604may determine that the health professional606is interested in learning about the injury and that disclosing such information via the second output data would be appropriate in the current circumstances. The portable electronic device604may actively and automatically output a second output data informing the health professional606and the user602regarding the injury. For example, the portable electronic device604may generate an output speech/response610indicating that “[name of user602] slipped last week landing on his arm. Here is how it happened.” The output speech/response610may further include details regarding the recorded event (i.e. accident) in order to output a second output data for providing proactive assistance for an improved diagnosis and/or for providing a reminder of when and how the injury occurred.

The portable electronic device604may output a first output data and/or a second output data via the output device114or various other devices or methods without limiting the scope of the present invention. For example, the portable electronic device604may output a second output data using the communications unit104.

In one embodiment, the portable electronic device604may be equipped with a display screen of displaying an output image/video614as determined by the processor102. For example, the portable electronic device604may have previously recorded the slip and fall accident leading to the arm injury when the accident occurred. If so, the portable electronic device604may output the second output data by displaying the output image/video614in order to provide assistance for the diagnosis, treatment, or consultation. In other embodiments, the second output data may be communicated to another device of the user602or the health professional606(e.g., another portable electronic device).

The source of the detected data may not necessarily be the sensors106. That is, multiple storage units and electronic devices may operate in concert sharing information in order to improve the quality and increase the quantity of information on which the outputted data are based. For example, the portable electronic device604may have access to the Internet or be in communication with the cloud-based database and computing110to receive input data previously or currently stored by an external device (e.g., a laptop computer) of the user602. For example, if the user602has previously stored data in any of the electronic devices in communication with the cloud-based database and computing110, the portable electronic device604may search such previously stored data for relevant information regarding the arm injury and output a second output data as described above accordingly.

Referring toFIG. 7, in one embodiment of the method/system700, the portable electronic device704may be the portable electronic device604discussed above with respect toFIG. 6. For example, the health professional706intends to provide recommendations as to courses of action that would improve a health condition of the user702. For example, the health professional706may utter the speech/recommendation708, stating that “I see how you got injured. Please maintain a nutritious diet including protein and vegetables, and perform weekly exercise/physical therapy routines of A, B, and C using 10 pound free weights.” The processor102of the portable electronic device704analyzes the elements of the speech/recommendation708. The method/system700stores and catalogues the information relating to exercise/physical therapy routines of A, B, and C using 10 pound free weights and the portable electronic device704and recognizes that the exercise/physical therapy routines of A, B, and C have to be performed once a week as instructed. The method/system700may automatically schedule reminders in the database112to be outputted on a weekly basis. The method/system700further stores and catalogues the analyzed data (e.g., regarding the recommended dietary regimens) in the database112.

Referring toFIG. 8, in one embodiment of the method/system800, the portable electronic device804may be the portable electronic device704discussed above with respect toFIG. 7. The portable electronic device804may recommend, motivate, or remind the user802of the information collected as discussed above with respect toFIGS. 6 and 7. For example, the method/system800may analyze the detected data in order to determine whether the user802is complying with the dietary and physical therapy guidelines set forth in the speech/recommendation708.

For example, based on the analyzed first output data, the method/system800may determine that the user802has only performed physical exercise routine A during the past week. The method/system800may further determine that the user802is watching the TV812and eating a certain type of food814(e.g., a cheesecake) using various object recognition detection and processing methods and systems. The method/system800may further recognize that the user802is currently resting on the couch, and has not performed a substantial degree of physical activities during the past week.

The method/system800may search the Internet, the cloud-based database and computing110, the memory108, or other data to determine the nutritional value of the food814. For example, the method/system800may generate the output speech808in order to encourage the user802to undertake a healthier food choice in order to facilitate a faster recovery based on the recommendations of the health professional706. The method/system800may further seek to motivate the user802to perform the physical therapy routines of B and C. The output speech808may be “A week has passed since you last performed physical therapy routines B and C. You can use the light weights in the room per health professional's instructions. There are vegetables and eggs in the fridge to help maintain a nutritious diet.” In another embodiment, the portable electronic device804may further remind the user802of taking medication properly if the health professional706had prescribed medication and provided instructions for taking the medication.

In other embodiments, the method/system800may be coupled to or in communication with a pedometer or other systems providing health information in order to have access to additional information regarding the health of the user802. Alternatively, the method/system800may determine health factors based on the detected data corresponding to exercise activities, dietary habits, and other matters with or without an external health monitoring device.

Referring toFIG. 9, in method/system900, the user902may suffer from Alzheimer's complications or dementia, or alternatively, the user902may have certain difficulties with retaining, retrieving, or analyzing information using human brain/memory. The method/system900may recognize that the user902is suffering from such complications based on the pre-programmed algorithm and/or detected data. In one embodiment, the method/system900may implicitly provide information that is appropriate under the circumstances in order to jog the memory of the user902. For example, a portable electronic device904may be continuously analyzing detected data within the proximity of the user902in order to determine if a new object, person, living being, or the other matter has entered the environment.

For example, as shown inFIG. 9, the portable electronic device904may recognize, using the detected data via the sensors106, that a person has entered the environment by opening the door914. The user902may utter the inquiry/thought908of “Who is that? He looks familiar.” If the user902does not utter the inquiry/thought908, the method/system900may recognize that the user902has not identified the newly entered person906. For example, the processor102of the portable electronic device904may utilize a camera to operate face recognition of at least a facial feature of the user902to determine that the user902is perplexed while attempting to identify the newly entered person906. In another embodiment, the method/system900may automatically output the second output data to jog the memory of the user902in a subtle manner as described below. Such an embodiment may be set as a default mode of operation if the portable electronic device is aware that the user902may have difficulties in retaining or retrieving information from memory.

For example, the processor102of the portable electronic device904may utilize a camera to operate face recognition of at least a facial feature of the newly entered person906. The method/system900may recognize that the newly entered person906is Michael, the nephew of the user902. In another embodiment, the recognition of the identity of the user902may be through speech recognition and the source of the speech (e.g., using a 3-D microphone). For example, if the newly entered person906utters a speech910, the processor102may determine the identity of the newly entered person906and look up prior relevant data regarding the newly entered person906using the method/system400. Other characteristics, interactions, physical appearance, or the like may be analyzed in order to determine the identity of the newly entered person906.

The processor102may take into account the reliability of a particular piece of information detected by one of the sensors106. For example, for identification or recognition of the newly entered person906, the processor102may assign various probabilities of accurate detection or recognition. For example, the assigned probabilities of accurate detection or recognition may be based on the mechanism used to detect the newly entered person906. In one embodiment, recognition or identification of the newly entered person906based on an identity code programmed in an identity device (e.g., a cellular phone) of the newly entered person906may not be as reliable as recognition based on a facial feature of the newly entered person906using a camera or speech recognition using a microphone, given that the newly entered906could be carrying an identity device of another individual.

The assigned probabilities of accurate identification or recognition may be further based on the circumstances under which the data is detected. For example, if a camera is obstructed causing the face of the newly entered person906to be unrecognizable, the method/system900may assign a relatively low probability of accurate detection or recognition in identifying the newly entered person906. If the obstruction is removed and a facial recognition processing mechanism confirms the identity of the newly entered person906, a higher probability of confidence would be associated with the identification or recognition. As such, the processor102may identify or recognize the newly entered person906based on a combination of various instances of data detection and further based on the corresponding probabilities of accurate recognition or identification.

Referring toFIG. 9, when the method/system900determines that the newly entered person906is “Michael,” a nephew of the user902, the portable electronic device904outputs appropriate information to provide indirect, subtle, and implicit reminders to the user902. For example, if the method/system900recognizes that the newly entered person906is a close relative, the processor102of the portable electronic device904may determine that outputting information regarding the last time that the newly entered person906and the user902have met would not be appropriate in the current circumstances. Instead, the method/system900may provide information in a subtle fashion to remind the user902that the newly entered person906is the nephew of the user902. For example, the portable electronic device904may direct the output reminder/speech912to the newly entered person, stating that “Hello Michael, your uncle [name of the user902] will be with you shortly,” thereby indirectly providing a reminder to the user902in a subtle fashion as appropriate under the circumstances.

Alternatively or in addition, the user902may request the portable electronic device904to generate additional information. The portable electronic device904may passively output, using the communications unit104, a first output data upon the request from the user902.

Referring to the method/system1000ofFIG. 10, in one embodiment, the vehicle1004may be a two-wheeled self-balancing device. The user1002may shift his/her weight in order to direct the vehicle1004to maneuver towards a desired direction. The handle bar of the vehicle1004may be included solely for providing additional comfort for the user1002given that the detected weight shift of the user1002using the base portion1014of the vehicle1004may contain sufficient maneuvering directions from the user1002.

In one embodiment, the vehicle1004may communicate directly with the user1002. For example, the vehicle1004may have speech recognition and generation capabilities as discussed above. For example, a camera may be implemented on the vehicle1004to capture first and/or data regarding the surrounding environment of the user1002. In another embodiment, the user1002may communicate with a portable electronic device of the user1002(e.g., the portable electronic device304,504,604,704,804, or904discussed above with respect toFIG. 3,5,6,7,8, or9, respectively) in communication with or coupled to a communication port of the vehicle1004. For example, a software application may be installed on a smart phone of the user1002in order to communicate data with the vehicle1004.

As shown inFIG. 10, the method/system1000has determined, based on the detected, processed, and stored data that the user1002seeks to perform physical activities and exercises in order to maintain a healthier lifestyle. The method/system1000may include or be in communication with a GPS device. The method/system1000may recognize that a building is located on the right hand side of the user1002. The method/system1000may automatically obtain data regarding the building and determine that the building is a gymnasium1006. The data may be obtained from the combination of data received from the GPS device and previously detected, processed, and stored data. The method/system1000may further determine the location of the home1010of the user1002and the distance and directions from the home1010of the user1002to the gymnasium1006. The method/system1000may automatically output a second output data recommending that the user1002consider signing up at the gymnasium1006.

The method/system1000may determine that based on the driving directions obtained from the GPS device, the route will take 20 minutes by travelling via the vehicle1004and will take 4 minutes by travelling via an automobile on the road1012. In order to facilitate the decision making process, the method/system1000may provide further information, for example, by generating the following output recommendation/speech1008:

“Since you mentioned that you seek to work out to lose weight, I have noticed that a gym has opened on your right recently. If you use this device, it will take you 20 minutes and if you drive westbound using road1012, it will take you 4 minutes to get there. It may be a good idea to exercise in the morning before work.”

Referring to the method/system1100ofFIG. 11, a vehicle1104may be a wheelchair or a three-wheeled transportation device. Any of the sensors106described above may be implemented on the vehicle1104. For example, the sensors106may be installed in sensor locations1110,1120,1124, and1126. The sensors106may be positioned at any other portion of the vehicle1104without limiting the scope of the invention. For example, a camera may be positioned at the sensor location1120in order to collect data regarding the surrounding environment. A camera, microphone, or other sensors106described above may also be installed at sensor location1110. Perimeter monitoring sensors and laser scanning sensors may be installed as sensor locations1124and1126. IMU's, gyroscopes, GPS system, and the like may further be mounted on the vehicle1104. The vehicle1104may further include a user interface1112to communicate with the user1102.

The output device114of vehicle1104may be a mechanical device having actuators capable of maneuvering the vehicle1104based on a third output data outputted by the processor102. For example, the vehicle1104may include a joystick1114that allows the user1102to maneuver the vehicle1104towards a desired direction. Alternatively or in addition, tactile data of the user1102may be detected.

For example, in the case of an Alzheimer's patient or a user1102with short term memory difficulties, if the user1102forgets a particular destination, the method/system1100may guide the wheelchair and the user1102to the destination. If the user1102is controlling the method/system1100, e.g., using the joystick1114as described above, a response such as a vibration may be outputted to alarm the user1102that the user1102is heading the wrong way or to further guide the user1102towards the desired destination. Vibration motors can further provide feedback to the user1102. Such vibration motors can be integrated in or coupled to an article of manufacture carried or worn by the user1102. For example, a belt1128or a vest may be coupled to the vehicle1104by which a vibration may provide information to the user1102.

Using the sensors106, the method/system1100may recognize that an orthopedic clinic1106is located on the left of the vehicle1104and user1102. Using the process described in details above (e.g., with respect to the method/system400ofFIG. 4), the method/system1100may match the detected data regarding the observed orthopedic clinic1106with a previously stored event (e.g., an arm injury discussed with respect toFIG. 6). Using the method/system400described above with respect toFIG. 4, the method/system1100may output a second output data for providing a reminder data and recommending that the user1102visits the orthopedic clinic1106in order to receive treatment before the injury exacerbates. For example, the method/system1100may generate an output speech1108as follows: “You can visit the Orthopedic Clinic on your left for the arm injury you sustained last week.”

Referring toFIGS. 1-11, one important aspect of the invention is that information can be shared between all or various combinations of the devices set forth above inFIGS. 1-12. For example, the portable electronic device304,504,604,704,804, or904discussed above with respect toFIG. 3,5,6,7,8, or9, respectively, along with the vehicles discussed above with respect toFIGS. 2A,2B,10, and11, and the external devices such as communication devices, cell phones, laptops, cameras, or other devices capable of communicating with or being coupled to the listed devices or units can share information, using the cloud-based database and computing110or the Internet, in order to access, store, and process the pool of the detected, processed, and stored data.

As used herein, the term “network” includes any cloud, cloud computing system or electronic communications system or method which incorporates hardware and/or software components. Communication among the parties may be accomplished through any suitable communication channels, such as, for example, a telephone network, an extranet, an intranet, Internet, point of interaction device (point of sale device, personal digital assistant (e.g., android, iPhone®, Palm Pilot®, Blackberry®), cellular phone, kiosk, and the like), online communications, satellite communications, off-line communications, wireless communications, transponder communications, local area network (LAN), wide area network (WAN), virtual private network (VPN), networked or linked devices, keyboard, mouse and/or any suitable communication or data input modality. Specific information related to the protocols, standards, and application software utilized in connection with the Internet is generally known to those skilled in the art and, as such, need not be detailed herein.

“Cloud” or “Cloud computing” includes a model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction. Cloud computing may include location-independent computing, whereby shared servers provide resources, software, and data to computers and other devices on demand.

The steps of a method or algorithm described in connection with the examples disclosed herein may be embodied directly in hardware, in a software module executed by the processor102, or in a combination of the two. A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor102such that the processor102can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor102. The processor102and the storage medium may reside in an Application Specific Integrated Circuit (ASIC).

As will be appreciated by one of ordinary skill in the art, the methods/systems100-1100may be embodied as a customization of an existing system, an add-on product, upgraded software, a stand-alone system, a distributed system, a method, a data processing system, a device for data processing, and/or a computer program product. Furthermore, the methods/systems100-1100may take the form of a computer program product on a non-transitory computer-readable storage medium having computer-readable program code means embodied in the storage medium. Any suitable computer-readable storage medium may be utilized, including hard disks, CD-ROM, optical storage devices, magnetic storage devices, and/or the like.