ADAPTIVE SENSORS TO ASSESS USER STATUS FOR WEARABLE DEVICES

A device is provided, including a frame that supports two eyepieces, a capacitive sensor mounted on the frame, an inertial measurement unit mounted on the frame, and a circuit component inside the frame, wherein the circuit component electrically couples the capacitive sensor and the inertial measurement unit with a processor and a memory inside the frame. A method for using the above device is also provided.

BACKGROUND

Field

The present disclosure is directed to sensors for wearable devices. More specifically, embodiments as disclosed herein are directed to adaptive sensors to assess user status and device status in headsets and smart glasses.

Related Art

In the field of wearable devices, sensors are critical to correctly assess a user command or status, and also to determine whether the device should be placed in active mode or in sleep mode, to save power. In the case of headsets and smart glasses, there are many different configurations that may be indicative of the device not being used, and thus a complex set of conditions and sensors is desirable to accurately assess the device status. Moreover, head movements of a user are typically indicative of precise user intentions or cognitive reactions. However, accurate sensors to assess user status and intentions and device status is lacking for headsets and smart glasses.

SUMMARY

In a first embodiment, a device includes a frame that supports two eyepieces, a capacitive sensor mounted on the frame, an inertial measurement unit mounted on the frame, and a circuit component inside the frame, wherein the circuit component electrically couples the capacitive sensor and the inertial measurement unit with a processor and a memory inside the frame.

In a second embodiment, a computer-implemented method includes receiving, from a contact or proximity sensor mounted on a frame of a headset, a contact signal above a first threshold value, receiving, from an inertial measurement unit mounted on the frame of the headset, an inertial signal indicative of an orientation of the headset relative to a vertical direction, and identifying a status of the headset as one of an active status or a sleep status, based on the contact signal and the inertial signal.

In a third embodiment, a non-transitory, computer-readable medium including instructions which, when executed by a processor, cause a computer to perform operations. The operations include receiving, from a contact or proximity sensor mounted on a frame of a headset, a contact signal above a first threshold value, receiving, from an inertial measurement unit mounted on the frame of the headset, an inertial signal indicative of an orientation of the headset relative to a vertical direction, identifying a status of the headset as one of an active status or a sleep status, based on the contact signal and the inertial signal, switching the status of the headset between the active status and the sleep status, based on the contact signal and the inertial signal, and wirelessly transmitting, via a communications module, a signal to a remote device indicative of the active status or the sleep status of the headset.

In another embodiment, a system includes a memory storing instructions and one or more processors configured to execute the instructions and cause the system to perform operations. The operations include to receive, from a contact or proximity sensor mounted on a frame of a headset, a contact signal above a first threshold value, to receive, from an inertial measurement unit mounted on the frame of the headset, an inertial signal indicative of an orientation of the headset relative to a vertical direction, and to identify a status of the headset as one of an active status or a sleep status, based on the contact signal and the inertial signal.

In other embodiment, a system includes a first means for storing instructions, and a second means for executing the instructions to cause the system to perform a method. The method includes receiving, from a contact or proximity sensor mounted on a frame of a headset, a contact signal above a first threshold value, receiving, from an inertial measurement unit mounted on the frame of the headset, an inertial signal indicative of an orientation of the headset relative to a vertical direction, and identifying a status of the headset as one of an active status or a sleep status, based on the contact signal and the inertial signal.

In one embodiment, a headset configured for virtual reality, mixed reality, or augmented reality applications includes cameras to capture the eye and face region of the user, which combined with contact, proximity signals from a Hall sensor, and inertial measurement signals, are able to identify a status of the headset.

These and other embodiments will be clear to one of ordinary skill in light of the following.

In the figures, elements having the same or similar reference numerals are associated with the same or similar attributes or features, unless explicitly stated differently, otherwise.

DETAILED DESCRIPTION

Headsets and smart glasses used in virtual reality (VR), augmented reality (AR) or mixed reality (MR) applications may be carried or left over a surface by users in multiple configurations, while not being activated. For example, a user may carry a headset over the hairline, or facing backwards. In some instances, the user may lay the smart glasses on a table (folded or not, facing down on a hard surface, and the like). It is desirable that the smart glass have hardware (e.g., sensors) and software to correctly identify these different configurations to set the device on a sleep mode to reduce power consumption. Likewise, it is expected that the device be rapidly activated once the user is ready to interact with the smart glass. Accordingly, in embodiments disclosed herein, devices such as smart glasses include highly sensitive capacitive touch sensors combined with accelerometers and other inertial measurement units (IMUs), to correctly identify user status and device status.

When the device is in active mode, highly sensitive IMU sensors may be used to identify a head shake, or nod, from the user, indicating a cognitive reaction to VR/AR/MR content displayed on the smart glass or headset. This could also work with audio-only glasses to control music or accept / reject a call, and with camera-only glasses to trigger capture by head gestures.

In some embodiments, single capacitive cells may be configured as one-point contact sensors in different parts of a smart glass to provide a verification signal for smart glass usage and proper wearing by the user. In addition, in some embodiments, a camera in the smart glass may be used to identify non-active use of the device (e.g., when the camera points to a ceiling, the back of the user’s head, or is blocked by a surface at close range).

FIG.1illustrates an architecture10including a smart glass100, and a wearable device102, coupled to one another, to a mobile device110(e.g., a smart phone), a remote server130and to a database152, according to some embodiments. All the devices in network10may communicate with one another via wireless communications and exchange a first dataset103-1. Dataset103-1may include a recorded video, audio, or some other file or streaming media. The user of wearable device102and headset100is also the owner or is associated with mobile device110. In some embodiments, smart glass100may directly communicate with remote server130, database152, or any other client device (e.g., a smart phone of a different user, and the like) via network150. Mobile device110may be communicatively coupled with remote server130and database152via network150, and transmit/share information, files, and the like with one another, e.g., dataset103-2and dataset103-3, hereinafter, collectively referred to as “datasets103.”

Smart glass100may include an augmented reality (AR) display107in at least one of two eyepieces105. In some embodiments, smart glass or headset100may include multiple sensors such as IMUs, gyroscopes and accelerometers121, barometers, magnetometers, ambient light sensors, proximity sensors, microphones124, cameras123, and capacitive sensors125, configured as contact interfaces for different parts of the user’s body/head.

Other contact sensors125may include a pressure sensor, a thermometer, and the like. In some embodiments, the thermometer may detect when a smart glass or headset is on the face of a person by detecting heat from the face. Smart glass100may also include a hinge sensor122, to determine the position configuration of the hinge that allows the temple in the smart glass frame to open/close for the user to wear/store the smart glasses. Hinge sensor122may include a hall sensor, a magnetic sensor, a capacitive sensor, or an optical sensor, such as an infrared (IR) sensor including a source and a detector in a sensing package.

In some embodiments, cameras123may include a forward-looking camera mounted on a frame109and configured to collect images of a forward view. In some embodiments, the image of a forward view may be an image of a blocking object (e.g., a hard surface), or some other irrelevant image such as a ceiling or the floor, or an image that remains static with no indication of motion for a pre-selected period of time. The above scenarios may be indicative that smart glass100is not being actively used and it may be desirable to switch it from an active mode to a sleep mode. In some embodiments, cameras123may include a backward-looking camera (e.g., an eye or face tracking camera) mounted on frame109and configured to collect an image of a portion of the face of user101, including one or both eyes. Accordingly, when the image from the face indicates that user101is not present, or it has both eyes shut for a pre-selected period of time, a processor112may determine that smart glass100may be desirably switched from an active mode to a sleep mode. In some embodiments, a signal provided by an eye tracking device may be used to infer whether the user is wearing smart glass100or not. For example, an eye tracking device may receive in a holographic optical element (HOE) combiner a reflection of a portion of the eyes of user101when illuminated by an infrared -IR- radiation.

In addition, smart glass100and/or mobile device110may include a memory circuit120storing instructions, and processor112is configured to execute the instructions to cause smart glass100and/or mobile device110to perform, at least partially, some of the steps in methods consistent with the present disclosure. In some embodiments, artificial intelligence (AI) algorithms may be used to train sensors and devices121,122,123,124, and125on the behavior of user101, thus improving detection accuracy. In some embodiments, smart glass100, mobile device110, server130, and/or database152may further include a communications module118enabling the device to wirelessly communicate with server130via network150. Smart glass100may thus download a multimedia online content (e.g., dataset103-1) from remote server130, to perform at least partially some of the operations in methods as disclosed herein. In some embodiments, memory120may include instructions to cause processor112to receive and combine signals from the IMU sensors121, hinge sensors122, microphones124, capacitive sensors125, and other contact sensors, avoid false positives, and better assess user intentions and commands when a touch signal is received.

In some embodiments, capacitive sensors125are configured to provide a contact signal indicative of a contact of frame109with a face of user101, and to detect proximity via fringe field detection, and IMU sensors121are configured to provide an orientation signal indicative of an orientation of two eyepieces105relative to a vertical position, and wherein processor112is configured to identify an active use of smart glass100based on the contact signal and the orientation signal. AR display107may be configured to provide an image to user101when processor112determines that user101intends to activate smart glass100based on a signal from capacitive sensor125and a signal from IMU sensor121. In some embodiments, IMU sensor121is configured to provide a first signal for a sideways swing of the head of user101and a second signal for an up-down swing of the head of user101, and processor112is configured to identify a negative response to an augmented reality content when receiving the first signal and a positive response to the augmented reality content when receiving the second signal.

In some embodiments, processor112is configured to switch smart glass100from an active mode into a sleep mode or from a sleep mode into an active mode, based on a first signal from capacitive sensor125and a second signal from IMU sensor121. In some embodiments, processor112may keep the system active but stop/play music or redirect audio from a call to another BT device or the host device (e.g., similar to an earbud). In some embodiments, processor112is configured to switch smart glass100from an active mode into a sleep mode or from a sleep mode into an active mode based on a signal from hinge sensor122, the signal indicative of one of a folding or an unfolding of a temple over frame109. In some embodiments, processor112is configured to switch smart glass100from an active mode into a sleep mode or from a sleep mode into an active mode, based on an image provided by forward-facing camera123mounted on frame109. In some embodiments, communications module118is electrically coupled with processor112, and is configured to wirelessly transmit a signal to a remote device inductive of an active status or a sleep status of the device (e.g., smart glass100, or wristband wearable102).

FIG.2illustrates an inactive smart glass200with temples203-1and203-2(hereinafter, collectively referred to as “temples203”) folded, according to some embodiments. Smart glass200includes an IMU sensor221, a camera223, a memory120, and a processor112, consistent with the present disclosure. In such configuration, hinges212-1and212-2(hereinafter, collectively referred to as “hinges212”) coupling frame209with temples203are closed (e.g., smart glass200is sitting on a table or counter, facing upward). To ensure users or bystanders feel comfortable with their privacy protection, processor112may put smart glass200into a sleep mode when hinge sensors222-1and222-2(hereinafter, collectively referred to as “hinge sensors222”) detect the closed status of one or two of the hinges212.

In some embodiments, hinge sensors222may include a hall sensor to detect a closed hinge212. In addition, in some embodiments, processor112may receive IMU data to further assess the inactivity of smart glass200. A communications module118transmits and receives data from an external device or network, as described above (cf. mobile device110, network150, server130).

FIG.3illustrates an inactive smart glass300face-down on a surface (or table)30, according to some embodiments. A camera323, an IMU sensor321, and one or two hinge sensors322-1and322-2(hereinafter, collectively referred to as “hinge sensors322”) may be mounted on frame309. Smart glass300also includes a processor312and a memory320, as disclosed herein.

Accordingly, camera323and IMU sensor321may provide dates with which processor312identifies when the user is wearing smart glasses300upside down or smart glasses300are sitting on table30, upside down. In such configurations, it may be desirable that smart glass300be set into a sleep mode, to avoid unnecessary power consumption. In addition, IMU sensor321may track tilt angle and record the state in memory320. Data from IMU sensor321stored during a capture of camera323, may be used for image post processing.

FIGS.4A-4Billustrate a user401wearing an inactive smart glass400, on the forehead (FIG.4A) or in the back of the head (FIG.4B), according to some embodiments. When user401is wearing smart glasses400on the forehead, or in the back of the head (facing backward), smart glass400is likely not in use, or inactive. In this configuration, user401usually does not expect smart glass400to be active and/or the device could be customized to user preferences with a gesture/movement trigger. Accordingly, processor412identifies this state based on a signal provided by IMU sensor421(e.g., indicating a tilt angle when user401has glasses400on the forehead, glasses400will be tilted slightly upwards), a capacitive sensor425, and/or a camera423, to put smart glass400to sleep. Capacitive sensor425may include an array (on left and right temple arms and frame). A memory420stores data and instructions for processor412to execute operations as disclosed herein.

FIG.5illustrates a user501nodding550-1/ shaking550-2(hereinafter, collectively referred to as “head gestures550”) its head as a cognitive response to an augmented reality input from a smart glass500, according to some embodiments. Smart glass500includes a camera523, a memory, a processor, and an IMU sensor521, consistent with the present disclosure. In some embodiments, when user501interacts with an AR application in smart glass500, user501prefers to use micro head movement to confirm yes or no to smart glass500. Such scenario may occur in a noisy restaurant or indoor situation where audio commands are buried by noise or in very quiet places when voice commands are not socially acceptable (e.g., class, church, an auditorium or concert hall, and the like).

Accordingly, the AR application in smart glass500may include user interface via head gestures550. For example, a nod550-2twice may be interpreted by processor512as a “YES,” and a left-to-right (or vice-versa) shake of the head550-2may be interpreted by processor512as a “NO.” A memory520stores instructions to be executed by sensor512. In some embodiments, the user interface may be assisted with gaze tracking and scene understanding and contextualization, to identify which object the user intends to interact with.

FIG.6illustrates a chart600indicating some of the power states602-1(‘sleep’),602-2(‘inactive’),602-3(‘ready’), and602-4(‘active,’ hereinafter, collectively referred to as “power states602”), of a smart glass and the transition thereof based on different sensor signals (610-1,610-2,610-3,610-4,610-5and610-6, collectively referred to as “sensor signals610”), according to some embodiments. In some embodiments, sleep state602-1may be the lowest power consumption state. Inactive state602-2may consume more power than sleep state602-1, as some radio receivers and sensors may be kept ‘on’ to set the device into ready state602-3, somewhat more power, or into an active state602-4(full power ‘on’). In the active state602-4, all sensors and radio transceivers are ‘on,’ and operating. For example, in active state602-4the camera may start collecting a video and an immersive reality application (VR/AR/MR) may be active as well.

Signal610-1leading to sleep state602-1from inactive state602-2may include a time lapse beyond a selected threshold during which the smart glass has been inactive. A signal610-3inducing sleep state602-1may include a lack of skin contact with the user’s face, or a camera picture of flat, featureless field indicative that the smart glass is facing a wall, a flat surface, the ceiling or the sky (cf.FIG.3). Accordingly, signal610-3may be indicative that the smart glass has been dropped, abandoned, forsaken or set away intentionally, by the user. When an opposite event occurs (the user finds the device and flips it on her/his face, a signal610-2may set the device to ready state602-3from sleep state602-1. From ready state602-3, the user may activate610-4the smart glass by pressing or touching a button in a contact sensor and set the device (e.g., a camera, or an immersive reality application) into active mode602-4. Active state602-4may be induced into inactive state602-2by the user pressing610-5a button in a contact sensor to deactivate or turn a device or sensor off (e.g., a video camera, or an immersive reality application). In some embodiments, active state602-4may transition into inactive state602-2when the sensors in the smart glass do not sense skin contact (but detect hair contact) or sense a rotation motion from an IMU indicative that the smart glass is placed in the forehead or flipped in the back of the user’s head (cf.FIGS.4A-4B). A signal610-6from contact sensors turning on the camera or an immersive reality application will turn the device from inactive state602-2to active state602-4. In some embodiments. A signal610-3from an IMU sensor indicative that the smart glass is back in the user’s face will set the smart glass from inactive state602-2to ready state602-3.

FIG.7is a flow chart illustrating steps in a method700for identifying a user command in a headset, according to some embodiments. In some embodiments, at least one or more of the steps in method700may be performed by a processor executing instructions stored in a memory in either one of a smart glass or other wearable device on a user’s body part (e.g., head, arm, wrist, leg, ankle, finger, toe, knee, shoulder, chest, back, and the like). In some embodiments, at least one or more of the steps in method700may be performed by a processor executing instructions stored in a memory, wherein either the processor or the memory, or both, are part of a mobile device for the user, a remote server or a database, communicatively coupled with each other via a network (cf. processors112,312,412, and512, memories120,320,420, and520, smart glasses100,200,300,400, and500, wristband102, mobile device110, server130, database152, and network150). Moreover, the mobile device, the smart glass, and the wearable devices may be communicatively coupled with each other via a wireless communication system and protocol (e.g., radio, Wi-Fi, Bluetooth, near-field communication -NFC- and the like as in communications module118). In some embodiments, a method consistent with the present disclosure may include one or more steps from method700performed in any order, simultaneously, quasi-simultaneously, or overlapping in time.

Step702includes receiving, from a contact sensor mounted on a frame of a headset, a contact signal above a first threshold value.

Step704includes receiving, from an inertial measurement unit mounted on the frame of the headset, an inertial signal indicative of an orientation of the headset relative to a vertical direction. In some embodiments, the inertial measurement unit is configured to provide a signal for a sideways swing of a user’s head, and step704further includes identifying a negative response to an augmented reality content when receiving the signal. In some embodiments, the inertial measurement unit is configured to provide a signal for an up-down swing of a user’s head, and step 604 further includes identifying a positive response to an augmented reality content when receiving the signal.

Step706includes identifying a status of the headset as one of an active status or a sleep status, based on the contact signal and the inertial signal. In some embodiments, step706further includes switching the status of the headset between the active status and the sleep status, based on the contact signal and the inertial signal. In some embodiments, step706further includes receiving, from a hinge detector, a signal indicative of a position configuration of a hinge joining a temple with the frame of the headset. In some embodiments, step706includes receiving, from a camera mounted on the frame of the headset, an image of a forward field of view of the headset. In some embodiments, the contact signal is indicative of a contact of the frame with a user’s face, and the inertial signal is indicative of an orientation of the headset relative to a vertical position, and step706includes verifying the contact signal and verifying that the orientation of the headset is substantially parallel to the vertical position. In some embodiments, step706includes switching the headset from an active mode into a sleep mode or from a sleep mode into an active mode based on a signal from a hinge sensor, the signal indicative of one of a folding or an unfolding of a temple over the frame. In some embodiments, step706includes switching the headset from an active mode into a sleep mode or from a sleep mode into an active mode, based on an image provided by a forward-facing camera mounted on the frame. In some embodiments, step706further includes wirelessly transmitting, via a communications module, a signal to a remote device indicative of the active status or the sleep status of the headset.

Hardware Overview

FIG.8is a block diagram illustrating an exemplary computer system800with which a VR, AR or MR headset, and methods of use can be implemented, according to some embodiments. In certain aspects, computer system800may be implemented using hardware or a combination of software and hardware, either in a dedicated server, or integrated into another entity, or distributed across multiple entities. Computer system800may include a desktop computer, a laptop computer, a tablet, a phablet, a smartphone, a feature phone, a server computer, or otherwise. A server computer may be located remotely in a data center or be stored locally.

Computer system800includes a bus808or other communication mechanism for communicating information, and a processor802coupled with bus808for processing information. By way of example, the computer system800may be implemented with one or more processors802. Processor802may be a general-purpose microprocessor, a microcontroller, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), a Programmable Logic Device (PLD), a controller, a state machine, gated logic, discrete hardware components, or any other suitable entity that can perform calculations or other manipulations of information.

Computer system800can include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them stored in an included memory804, such as a Random Access Memory (RAM), a flash memory, a Read-Only Memory (ROM), a Programmable Read-Only Memory (PROM), an Erasable PROM (EPROM), registers, a hard disk, a removable disk, a CD-ROM, a DVD, or any other suitable storage device, coupled with bus808for storing information and instructions to be executed by processor802. The processor802and the memory804can be supplemented by, or incorporated in, special purpose logic circuitry.

Computer system800further includes a data storage device806such as a magnetic disk or optical disk, coupled with bus808for storing information and instructions. Computer system800may be coupled via input/output module810to various devices. Input/output module810can be any input/output module. Exemplary input/output modules810include data ports such as USB ports. The input/output module810is configured to connect to a communications module812. Exemplary communications modules812include networking interface cards, such as Ethernet cards and modems. In certain aspects, input/output module810is configured to connect to a plurality of devices, such as an input device814and/or an output device816. Exemplary input devices814include a keyboard and a pointing device, e.g., a mouse or a trackball, by which a consumer can provide input to the computer system800. Other kinds of input devices814can be used to provide for interaction with a consumer as well, such as a tactile input device, visual input device, audio input device, or brain-computer interface device. For example, feedback provided to the consumer can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the consumer can be received in any form, including acoustic, speech, tactile, or brain wave input. Exemplary output devices816include display devices, such as an LCD (liquid crystal display) monitor, for displaying information to the consumer.

According to one aspect of the present disclosure, a VR headset as disclosed herein can be implemented, at least partially, using a computer system800in response to processor802executing one or more sequences of one or more instructions contained in memory804. Such instructions may be read into memory804from another machine-readable medium, such as data storage device806. Execution of the sequences of instructions contained in main memory804causes processor802to perform the process steps described herein. One or more processors in a multi-processing arrangement may also be employed to execute the sequences of instructions contained in memory804. In alternative aspects, hard-wired circuitry may be used in place of or in combination with software instructions to implement various aspects of the present disclosure. Thus, aspects of the present disclosure are not limited to any specific combination of hardware circuitry and software.

The subject technology is illustrated, for example, according to various aspects described below. Various examples of aspects of the subject technology are described as numbered claims (claim 1, 2, etc.) for convenience. These are provided as examples and do not limit the subject technology.

While this specification contains many specifics, these should not be construed as limitations on the scope of what may be described, but rather as descriptions of particular implementations of the subject matter. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable sub-combination. Moreover, although features may be described above as acting in certain combinations and even initially described as such, one or more features from a described combination can in some cases be excised from the combination, and the described combination may be directed to a sub-combination or variation of a sub-combination.