Head mounted device and camera module

Devices and camera modules are configured for imaging facial gestures. The imaging may be directed at lower face imaging. The camera modules may be disposed along outside surfaces of devices such as head mounted devices to have a field of view that images a lower face region.

TECHNICAL FIELD

This disclosure relates generally to optics, and in particular to imaging.

BACKGROUND INFORMATION

Camera modules are included in smartphones, televisions, laptops, and other devices to capture images. Camera modules typically include an image sensor and one or more lenses configured to focus visible image light onto the image sensor. A display of the device may then present the image to a user of the device.

DETAILED DESCRIPTION

In some implementations of the disclosure, the term “near-eye” may be defined as including an element that is configured to be placed within 50 mm of an eye of a user while a near-eye device is being utilized. Therefore, a “near-eye optical element” or a “near-eye system” would include one or more elements configured to be placed within 50 mm of the eye of the user.

In aspects of this disclosure, visible light may be defined as having a wavelength range of approximately 380 nm-700 nm. Non-visible light may be defined as light having wavelengths that are outside the visible light range, such as ultraviolet light and infrared light. Infrared light having a wavelength range of approximately 700 nm-1 mm includes near-infrared light. In aspects of this disclosure, near-infrared light may be defined as having a wavelength range of approximately 700 nm-1.6 μm.

In aspects of this disclosure, the term “transparent” may be defined as having greater than 90% transmission of light. In some aspects, the term “transparent” may be defined as a material having greater than 90% transmission of visible light.

Head mounted devices such as augmented reality (AR) or virtual reality (VR) headsets present virtual images to users. Head mounted devices are configured to be worn on or about a head of a user and head mounted devices may optionally include a display such that the head mounted device is considered a head mounted display (HMD). In some head mounted devices, cameras have been included to capture images of the external environment of the head mounted device. Recently, cameras have been included in head mounted devices to image the eye of a user. Imaging the eye of a user may allow the user to interact with virtual images or control an operation of the head mounted device.

Facial gestures by a user may also increase interaction with a device or a community using a head mounted device. By way of example, imaging facial gestures may allow real-time animation of a user's facial gesture to be shared with other users. For example, an avatar of a user of a head mounted device may be animated to smile in response to imaging a user that is smiling. Of course, other facial gestures can be translated into animation of an avatar. Furthermore, imaging facial gestures may allow for other interactions and may also be used as an input to control the operation of a head mounted device.

Implementations of the disclosure include camera modules that may be configured for imaging facial gestures. In an implementation, a lower-face imaging camera is included in a head mounted device. In some implementations, the head mounted device includes a display for presenting virtual images to users of the device and the head mounted device is considered an HMD. The lower-face imaging camera module may be configured to image a lower face region that includes a lip region, a chin region, and/or a lower-nose region. Imaging the lip, chin, and/or lower-nose region may assist in determining facial gestures. The lower-face imaging camera module may be located on an outside-bottom surface of a head mounted device to orient the camera module to have a field-of-view (FOV) that includes the lower face region. In some implementations, non-visible light sources (e.g. infrared light sources) emit non-visible illumination light to illuminate the lower face region and the lower-face imaging camera module is configured to image the wavelength of the non-visible light. In some implementations, more than one lower-face imaging camera modules may be used to image the facial gestures.

In an implementation, a head mounted devices includes a transparent cover that covers a camera of the head mounted device. A total internal reflection (TIR) suppression bracket is included in the head mounted device where the TIR suppression bracket is angled with respect to the transparent cover to reflect image light incident onto the TIR suppression bracket at a reflection angle that exits the transparent cover without the transparent cover confining reflected image light by way of TIR. This reduces optical noise (that may cause ghost images) from entering the camera of the head mounted device. In some implementations, the TIR suppression bracket also prevents dust from contacting the camera. In some implementations, an energy-absorbing mechanical structure is coupled to support an image sensor and a lens assembly of a camera module and the energy-absorbing mechanical structure is configured to be secured to a chassis of the head mounted device. These and other embodiments are described in more detail in connection withFIGS.1-5B.

FIG.1is a perspective view of a head mounted display (HMD)100configured for imaging facial gestures, in accordance with an implementation of the disclosure. The illustrated example of HMD100is shown as including a viewing structure140including a front rigid body144, a top securing structure141, a side securing structure142, and a rear securing structure143. In some examples, the HMD100is configured to be worn on a head of a user of the HMD100, where the top securing structure141, side securing structure142, and/or rear securing structure143may include a fabric strap including elastic as well as one or more rigid structures (e.g., plastic) for securing the HMD100to the head of the user. HMD100may also optionally include one or more earpieces120for delivering audio to the ear(s) of the user of the HMD100.

The illustrated example of HMD100also includes an interface membrane118for contacting a face of the user of the HMD100, where the interface membrane118functions to block out at least some ambient light from reaching to the eyes of the user of the HMD100.

Example HMD100may also include a chassis for maintaining mechanical structures of HMD100and for supporting hardware of the viewing structure140of HMD100(chassis and hardware not explicitly illustrated inFIG.1). The hardware of viewing structure140may include any of processing logic, wired and/or wireless data interface for sending and receiving data, graphic processors, and one or more memories for storing data and computer-executable instructions. In one example, viewing structure140may be configured to receive wired power and/or may be configured to be powered by one or more batteries. In addition, viewing structure140may be configured to receive wired and/or wireless data including video data.

Viewing structure140may also include a display system having one or more electronic displays for directing image light to the eye(s) of a user of HMD100to present a virtual image to the user. The display system may include one or more of an LCD, an organic light emitting diode (OLED) display, or micro-LED display for emitting image light (e.g., content, images, video, etc.) to a user of HMD100.

FIGS.2A-2Cillustrate a user wearing HMD100that includes a lower-face imaging camera module for imaging facial gestures, in accordance with implementations of the disclosure.FIG.2Aillustrates a user wearing HMD100so that the user can view virtual images presented by HMD100.FIG.2Billustrates example placement of a display160and display optics165that may be configured to focus virtual images to the eye of the user.

FIG.2Cillustrates a zoomed-in view ofFIG.2Bthat includes a lower-face imaging camera module233configured to image a lower face region of a user in a FOV256of lower-face imaging camera module233. Lower-face imaging camera module233may include an image sensor and a lens assembly configured to focus image light onto the image sensor. The image sensor may be a complementary metal-oxide semiconductor (CMOS) image sensor, for example.

HMD100may include a plurality of lower-face imaging camera modules233, in some implementations. In the illustrated implementation, lower-face imaging camera module233is located at an outside-bottom surface239of the viewing structure140of HMD100. This may provide the lower-face imaging camera module233the required angle to image the lower face region of a wearer of HMD100. Outside-bottom surface239may be tilted down and facing the lower face region when a user has HMD100positioned to view the virtual images presented on the display, as illustrated inFIG.2C.

A lens assembly of lower-face imaging camera module233may have a focus distance between 0.5 cm and 5 cm to allow the lens assembly to focus features of the lower face region (e.g. lips, chin, and/or nose) to the image sensor of lower-face imaging camera module233. In some implementations, the lens assembly has a focus distance of between 0.5 cm and 10 cm. In some implementations, the lens assembly has a focus distance of between 2.5 cm and 10 cm. In an implementation, the lens assembly has a focus distance between 16 mm and 40 mm.

FIG.2Calso illustrates that a non-visible light source237may be oriented to emit non-visible illumination light238to illuminate the lower face region. The non-visible light source237may be an LED, micro-LED, vertical-cavity side-emitting laser (VCSEL), or other suitable light source. The non-visible light source237may emit infrared light. The non-visible light source may emit near-infrared light. The lower-face imaging camera module233may be configured to image a near-infrared wavelength band of the near-infrared light and a filter of the lower-face imaging camera module may be configured to block wavelengths outside the near-infrared wavelength band from becoming incident on an image sensor of the lower-face imaging camera module233. For example, non-visible light source237may be configured to emit near-infrared light centered around 940 nm and the filter of lower-face imaging camera module233may be configured to transmit light having a wavelength between 935 nm and945while rejecting (blocking) other wavelengths.

FIG.3Aillustrates an example lower face region356that may be imaged by lower-face imaging camera module233, in accordance with implementations of the disclosure. Hence, lower face region356may be an example of what is in FOV256of lower-face imaging camera module233. Example lower face region356includes a lower-nose region374, a lip region376, and a chin region378. These regions may overlap in some implementations. HMD100may cover an upper face region of a user (e.g. above lower-nose region374) so imaging the lower face region356may be particularly important for determining facial gestures of a user. For example, an upper-nose region above lower-nose region374may be outside FOV256and thus outside of lower face region356.

Lip region376may include a region where lips375of the user/wearer will occupy when a wearer of HMD100is wearing HMD100(or other head mounted device) on their head. Lip region376may include lips375of a user as well as areas around the lips so that smile lines, dimples, etc. can be imaged. Imaging lips375, and surrounding skin may be important for detecting gestures or reactions of a user, for example. In the illustrated ofFIG.3A, chin region378includes a region where a chin377of a user will occupy when the user is wearing HMD100. Chin region378may extend from a bottom of chin379to a bottom of lips375. In some implementations, lower face region356does not include chin region378. In some implementations, lower face region356does not include lower-nose region374that includes a nose373of a user of HMD100(or other head mounted device). In some implementations, lower face region356does not include chin region378.

FIG.3Billustrates that lower face region356may be imaged by more than one lower-face imaging camera module233, in some implementation. By way of example,FIG.3Billustrates that a first camera module may image a left side361of lower face region356and a second camera module may image a right side362of lower face region356. Left side361of lower face region356is depicted with a dash-dot line and right side362of lower face region356is depicted with a mini dash line, inFIG.3B. In the illustrated example, left side361of lower face region356overlaps right side362of lower face region356in overlapping region363. In other implementations, more than two lower-face imaging camera modules may image lower face region356.

FIG.3Cillustrates a larger view of lower-nose region374, in accordance with implementations of the disclosure.FIG.3Cillustrates points382,383,384,385, and386as example anchor points to assist in image analysis of lower-nose region374.

FIG.4illustrates an interior view of example HMD400that includes a first lower-face imaging camera module433A and a second lower-face imaging camera module433B, in accordance with implementations of the disclosure. First lower-face imaging camera module433A may be configured to image left side361of lower face region356and second lower-face imaging camera module433B may be configured to image right side362of lower face region356, for example. In the illustration ofFIG.4, first lower-face imaging camera module433A and second lower-face imaging camera module433B are disposed on an outside-bottom surface439of the viewing structure440of HMD400. This may provide the lower-face imaging camera modules433(referring to modules433A and433B collectively) the required angle to image the lower face region of a wearer of HMD400. Outside-bottom surface439may be tilted down and facing the lower face region when a user has HMD400positioned to view the virtual images presented on the display.

The interior view of HMD400illustrates that display optical elements423A and423B that are configured to focus virtual images presented by a display (not specifically illustrated inFIG.4). Display optical elements423A and423B are surrounded by eyecups427A and427B, respectively. Eyecups427A and427B may be configured to block out ambient light from eyes of a user wearing HMD400, for example. InFIG.4, a nose void425of viewing structure440is disposed between first lower-face imaging camera module433A and second lower-face imaging camera module433B so that a nose of a wearer does not occlude (or reduces occlusions of) lower face region356. In implementations of the disclosure, non-visible light sources (e.g. similar to light source237) may be disposed near each lower-face imaging camera module433to illuminate lower face region356with non-visible illumination light.

FIG.5Aillustrates an example configuration500of a camera module of a head mounted device. Configuration500includes a camera module501that includes a lens assembly505and an image sensor503. Camera module501is mounted to mount515. Electrical components of camera module501may be routed to a printed circuit board (PCB) or a flex circuit of mount515. Dust bracket517is coupled to chassis513of the head mounted device. Dust bracket517prevents dust from encountering camera module501and, in particular, prevents dust from entering the space between cover511and lens assembly505so that dust or other contaminants do not encounter the imaging optical path of camera module501.

FIG.5Billustrates an example configuration550of a camera module551of a head mounted device, in accordance with implementations of the disclosure. Configuration550includes a camera module551that includes a lens assembly555and an image sensor553. Lens assembly555is configured to focus image light from FOV560onto image sensor553. Image sensor553may be configured to capture near-infrared images. Configuration550may be used in conjunction with the implementation described with respect toFIGS.1-4.

Camera module551is mounted to energy-absorbing mechanical structure565. Energy-absorbing mechanical structure565is coupled to support image sensor553and lens assembly555. The energy-absorbing mechanical structure565is configured to be secured to chassis563of a viewing structure of the head mounted device. A fastener such as fastener587may secure energy-absorbing mechanical structure565to the chassis (not specifically illustrated as attached to fastener587), for example.

In the illustration ofFIG.5B, energy-absorbing mechanical structure565includes a spring567to absorb impacts. Since camera module551may be mounted on a very outside edge/surface and/or a corner of the head mounted device (in order to get the appropriate angle to image facial gestures), energy-absorbing mechanical structure565may be particularly helpful to absorb mechanical shock rather than transferring the mechanical shock to the camera module551. Dropping head mounted devices is one concerning source of potential mechanical shock, for example. Electrical components of camera module551may be routed to a printed circuit board (PCB) or a flex circuit of energy-absorbing mechanical structure565.

Configuration550includes a transparent cover561secured to a viewing structure of the head mounted device by way of chassis563. Lens assembly555is disposed between image sensor553and transparent cover561. Total internal reflection (TIR) suppression bracket583is angled with respect to transparent cover561to reflect the image light (e.g. image light ray575) incident onto TIR suppression bracket583at a reflection angle that exits transparent cover561without transparent cover561confining reflected image light576by way of TIR. In contrast, dust bracket517ofFIG.5Aadds optical noise to images captured by image sensor503by reflecting incident image light at an angle conducive to cover511confining the reflected image light into a refractive material of cover511. Since the reflected image light is then confined to cover511by TIR, cover511may act as a waveguide that transfers reflected image light into lens assembly505when the waveguided light escapes TIR by cover511. This light that escapes TIR then is focused by lens assembly505onto image sensor503, which adds optical noise (e.g. ghosting artifacts) to images generated by image sensor503.

Energy-absorbing mechanical structure565is coupled to TIR suppression bracket583, inFIG.5B. TIR suppression bracket583is also coupled to chassis563of a device. Advantageously, in some implementations, TIR suppression bracket583is also configured to prevent dust from contacting lens assembly555and image sensor553(in addition to suppressing TIR). Transparent cover561may conform to the viewing structure of the head mounted device.

The term “processing logic” in this disclosure may include one or more processors, microprocessors, multi-core processors, Application-specific integrated circuits (ASIC), and/or Field Programmable Gate Arrays (FPGAs) to execute operations disclosed herein. In some embodiments, memories (not illustrated) are integrated into the processing logic to store instructions to execute operations and/or store data. Processing logic may also include analog or digital circuitry to perform the operations in accordance with embodiments of the disclosure.

A “memory” or “memories” described in this disclosure may include one or more volatile or non-volatile memory architectures. The “memory” or “memories” may be removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, or other data. Example memory technologies may include RAM, ROM, EEPROM, flash memory, CD-ROM, digital versatile disks (DVD), high-definition multimedia/data storage disks, or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information for access by a computing device.

Networks may include any network or network system such as, but not limited to, the following: a peer-to-peer network; a Local Area Network (LAN); a Wide Area Network (WAN); a public network, such as the Internet; a private network; a cellular network; a wireless network; a wired network; a wireless and wired combination network; and a satellite network.

Communication channels may include or be routed through one or more wired or wireless communication utilizing IEEE 802.11 protocols, BlueTooth, SPI (Serial Peripheral Interface), I2C (Inter-Integrated Circuit), USB (Universal Serial Port), CAN (Controller Area Network), cellular data protocols (e.g. 3G, 4G, LTE, 5G), optical communication networks, Internet Service Providers (ISPs), a peer-to-peer network, a Local Area Network (LAN), a Wide Area Network (WAN), a public network (e.g. “the Internet”), a private network, a satellite network, or otherwise.

A computing device may include a desktop computer, a laptop computer, a tablet, a phablet, a smartphone, a feature phone, a server computer, or otherwise. A server computer may be located remotely in a data center or be stored locally.