Mobile device holographic calling with front and back camera capture

Aspects of the present disclosure are directed to a holographic calling system providing holographic calling between an artificial reality device and a mobile device having both front and back facing cameras. The user of the mobile device can position it so one of the cameras is pointed toward their face and another camera captures the user's hand not holing the mobile device. The holographic calling system can automatically determine the position of the mobile device in relation to the user's face and hand. Once the mobile device is positioned within an appropriate capture zone, the captured images of the user's face are used to create a first representation of the sending user's face, the captured images of the user's hand are used to create a second representation of the sending user's hand. Each representation is provided as output from a receiving artificial reality device, positioned relative to each other.

TECHNICAL FIELD

The present disclosure is directed to implementing a holographic call between an artificial reality device and mobile device using both the mobile device's front and back facing cameras.

BACKGROUND

Video conferencing has become a major way people connect. From work calls to virtual happy hours, webinars to online theater, people feel more connected when they can see othe participants, bringing them closer to an in-person experience. However, video calls remain a pale imitation of face-to-face interactions. Understanding body language and context can be difficult with only a two-dimensional (“2D”) representation of a sender. Further, interpersonal interactions with video are severely limited as communication often relies on relational movements between participants. With video calling, participants are unable to perform movements in relation to one another. In addition, the limitation of video calling on a flat panel display introduces an intrusive layer of technology that can distract from communication and diminishes the perception of in-person communication.

DETAILED DESCRIPTION

Aspects of the present disclosure are directed to a holographic calling system providing holographic calling between an artificial reality device and a mobile device having both front and back facing cameras. The user of the mobile device can position it so one of the cameras is pointed toward their face and another camera captures the user's hand not holing the mobile device. The holographic calling system captures images of portions of the user from both the front-facing and back-facing cameras, providing different views of portions of the sending user (a facial/upper torso portion and a hand/arm portion). The holographic calling system can automatically determine the position of the mobile device in relation to the user's face and hand and, if necessary, instruct the user on repositioning the mobile device and/or her hand. Once the mobile device is positioned within an appropriate capture zone, the captured images of the user's face are used to create a first representation of the sending user's face, the captured images of the user's hand are used to create a second representation of the sending user's hand, and each are provided as output from a receiving artificial reality device, positioned relative to each other based on the determined hand and face positions of the sending user.

Depending on the capabilities of the mobile device the mobile device may simultaneously capture images from the front and back facing cameras or may alternate between enabling each camera. In some implementations where the mobile device has an array of multiple cameras facing one direction and/or a depth sensor facing either direction, the mobile device may also directly capture depth data for the captured images. In other implementations, a machine learning model may be applied to estimate depth data for captured images. In yet further implementations, a user may be wearing a device such as a smart bracelet or ring on her hand, and/or glasses or earphones on her face and distances between the mobile device and these devices can be measured (e.g., based on travel time for signals traveling at as a known velocity). In some implementations, the distances can be refined using a kinematic model defining user arm-span lengths. Based on these distance determinations, the holographic calling system can determine whether the mobile device is positioned in a capture zone, i.e., is within a threshold distance of the midpoint between the user's hand and face, there is at least a minimum distance between the mobile device and the user's hand and at least a minimum distance between the mobile device and the user's face, and the mobile device is correctly angled to capture each of the user's hand and face.

When the mobile device is not correctly positioned, the holographic calling system can determine how the mobile device should be moved to put it within the capture zone. The holographic calling system can then provide affordances to instruct the user on how to move the mobile device and/or her hand. These affordances can, for example, include arrows displayed on the mobile device screen, adding blurring and focus filters to the mobile device screen, providing a silhouette or other indicator of where to move the mobile device, etc.

Once the mobile device position has been positioned correctly, either initially or following instruction to the user through affordances, the holographic calling system can generate sending user representations. In some implementations, this can include applying the depth and other position information to a kinematic model (either generic or user-specific) to determine body positions of the sending user. In other implementations, the body positions of the sending user can be determined by directly estimating body positions from the captured depth data. In some cases, the body positions can then be used to generate an avatar representation of the sending user (which may be life-like from scans of the sending user or a synthetic model) with the determined body positions. In other implementations, the holographic calling system can generate a first holographic representation of the sending user by generating a hologram of the user's face and torso from the images of the camera facing those body parts, can generate a second holographic representation of the sending user by generating a hologram of the user's hand and arm from the images of the camera facing those body parts, and can have the output from an artificial reality device position each relative to each other according to the determined body positions.

When creating the holograms, the holographic calling system may extrapolate portions of the sending user not shown in the captured images. For example, a sending user may be holding up her hand with her palm facing away from the sending user. The recipient user may be viewing the hologram of the sending user's hand from the opposite side as the mobile device (i.e., the mobile device captures the back of the user's hand, but the recipient user sees the front of the sending user's palm). Thus the holographic calling system may generate a hand hologram that positions the hand according to the determined hand position of the sending user, but paints a generic palm texture onto the portion of the hand the mobile device did not capture.

“Virtual reality” or “VR,” as used herein, refers to an immersive experience where a user's visual input is controlled by a computing system. “Augmented reality” or “AR” refers to systems where a user views images of the real world after they have passed through a computing system. For example, a tablet with a camera on the back can capture images of the real world and then display the images on the screen on the opposite side of the tablet from the camera. The tablet can process and adjust or “augment” the images as they pass through the system, such as by adding virtual objects. “Mixed reality” or “MR” refers to systems where light entering a user's eye is partially generated by a computing system and partially composes light reflected off objects in the real world. For example, a MR headset could be shaped as a pair of glasses with a pass-through display, which allows light from the real world to pass through a waveguide that simultaneously emits light from a projector in the MR headset, allowing the MR headset to present virtual objects intermixed with the real objects the user can see. “Artificial reality,” “extra reality,” or “XR,” as used herein, refers to any of VR, AR, MR, or any combination or hybrid thereof.

While there are existing visual communication systems that use mobile devices, they tend to only provide flat-panel types of communications (e.g., video calls), which fail to enable communications comparable to in-person interactions. Some holographic calling systems are being developed, however they tend to require both the sending user and the recipient user to have specialized artificial reality device hardware, limiting which users can employ these technologies. For example, existing video calling systems limit how much body language can be perceived, fail to provide the ability for users to move relative to each other, and introduce an intrusive layer of technology that can distract from the communication and diminish the perception of in-person communication. Further, existing holographic calling technologies only allow holographic calls with other users having the same holographic system, limiting adoption, presenting extreme cost, and failing to allow existing hardware to interface with the artificial reality devices.

The holographic calling system and processes described herein are expected to overcome these problems associated with conventional video and 3D interaction techniques and are expected provide holographic calling between users where at least one of whom is using a mobile device (i.e., smartphone or tablet). By including techniques that can utilize the capabilities of the available mobile devices, the holographic calling system can allow user interactions beyond the capabilities of existing systems. In addition, the processes and systems to achieve these results are not analogs of existing communication techniques, but instead introduce completely new ways capturing multiple views of a user from a single mobile device and representing those multiple views relationally in a holographic call. Thus, unlike the existing video calling techniques that capture a single mobile device user view, the disclosed holographic calling system can generate holographic representations from a mobile device, can account for user movement and positions, and can format data to allow a recipient user to view multiple holograms of the sending user, correctly positioned relative to one other.

Several implementations are discussed below in more detail in reference to the figures.FIG.1is a block diagram illustrating an overview of devices on which some implementations of the disclosed technology can operate. The devices can comprise hardware components of a computing system100that can implement a holographic call between an artificial reality device and mobile device using both the mobile device's front and back cameras. In various implementations, computing system100can include a single computing device103or multiple computing devices (e.g., computing device101, computing device102, and computing device103) that communicate over wired or wireless channels to distribute processing and share input data. In some implementations, computing system100can include a stand-alone headset capable of providing a computer created or augmented experience for a user without the need for external processing or sensors. In other implementations, computing system100can include multiple computing devices such as a headset and a core processing component (such as a console, mobile device, or server system) where some processing operations are performed on the headset and others are offloaded to the core processing component. Example headsets are described below in relation toFIGS.2A and2B. In some implementations, position and environment data can be gathered only by sensors incorporated in the headset device, while in other implementations one or more of the non-headset computing devices can include sensor components that can track environment or position data.

Computing system100can include one or more processor(s)110(e.g., central processing units (CPUs), graphical processing units (GPUs), holographic processing units (HPUs), etc.) Processors110can be a single processing unit or multiple processing units in a device or distributed across multiple devices (e.g., distributed across two or more of computing devices101-103).

Computing system100can include one or more input devices120that provide input to the processors110, notifying them of actions. The actions can be mediated by a hardware controller that interprets the signals received from the input device and communicates the information to the processors110using a communication protocol. Each input device120can include, for example, a mouse, a keyboard, a touchscreen, a touchpad, a wearable input device (e.g., a haptics glove, a bracelet, a ring, an earring, a necklace, a watch, etc.), a camera (or other light-based input device, e.g., an infrared sensor), a microphone, or other user input devices.

In some implementations, input from the I/O devices140, such as cameras, depth sensors, IMU sensor, GPS units, LiDAR or other time-of-flights sensors, etc. can be used by the computing system100to identify and map the physical environment of the user while tracking the user's location within that environment. This simultaneous localization and mapping (SLAM) system can generate maps (e.g., topologies, girds, etc.) for an area (which may be a room, building, outdoor space, etc.) and/or obtain maps previously generated by computing system100or another computing system that had mapped the area. The SLAM system can track the user within the area based on factors such as GPS data, matching identified objects and structures to mapped objects and structures, monitoring acceleration and other position changes, etc.

Computing system100can include a communication device capable of communicating wirelessly or wire-based with other local computing devices or a network node. The communication device can communicate with another device or a server through a network using, for example, TCP/IP protocols. Computing system100can utilize the communication device to distribute operations across multiple network devices.

The processors110can have access to a memory150, which can be contained on one of the computing devices of computing system100or can be distributed across of the multiple computing devices of computing system100or other external devices. A memory includes one or more hardware devices for volatile or non-volatile storage, and can include both read-only and writable memory. For example, a memory can include one or more of random access memory (RAM), various caches, CPU registers, read-only memory (ROM), and writable non-volatile memory, such as flash memory, hard drives, floppy disks, CDs, DVDs, magnetic storage devices, tape drives, and so forth. A memory is not a propagating signal divorced from underlying hardware; a memory is thus non-transitory. Memory150can include program memory160that stores programs and software, such as an operating system162, holographic calling system164, and other application programs166. Memory150can also include data memory170that can include, e.g., images captured from front and back facing cameras on a mobile device, position data for a mobile device and a user's body parts, kinematic models, content item for affordances, holograms generated from front and back facing mobile device cameras, configuration data, settings, user options or preferences, etc., which can be provided to the program memory160or any element of the computing system100.

FIG.2Ais a wire diagram of a virtual reality head-mounted display (HMD)200, in accordance with some embodiments. The HMD200includes a front rigid body205and a band210. The front rigid body205includes one or more electronic display elements of an electronic display245, an inertial motion unit (IMU)215, one or more position sensors220, locators225, and one or more compute units230. The position sensors220, the IMU215, and compute units230may be internal to the HMD200and may not be visible to the user. In various implementations, the IMU215, position sensors220, and locators225can track movement and location of the HMD200in the real world and in an artificial reality environment in three degrees of freedom (3DoF) or six degrees of freedom (6DoF). For example, the locators225can emit infrared light beams which create light points on real objects around the HMD200. As another example, the IMU215can include e.g., one or more accelerometers, gyroscopes, magnetometers, other non-camera-based position, force, or orientation sensors, or combinations thereof. One or more cameras (not shown) integrated with the HMD200can detect the light points. Compute units230in the HMD200can use the detected light points to extrapolate position and movement of the HMD200as well as to identify the shape and position of the real objects surrounding the HMD200.

The electronic display245can be integrated with the front rigid body205and can provide image light to a user as dictated by the compute units230. In various embodiments, the electronic display245can be a single electronic display or multiple electronic displays (e.g., a display for each user eye). Examples of the electronic display245include: a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, an active-matrix organic light-emitting diode display (AMOLED), a display including one or more quantum dot light-emitting diode (QOLED) sub-pixels, a projector unit (e.g., microLED, LASER, etc.), some other display, or some combination thereof.

In some implementations, the HMD200can be coupled to a core processing component such as a personal computer (PC) (not shown) and/or one or more external sensors (not shown). The external sensors can monitor the HMD200(e.g., via light emitted from the HMD200) which the PC can use, in combination with output from the IMU215and position sensors220, to determine the location and movement of the HMD200.

FIG.2Bis a wire diagram of a mixed reality HMD system250which includes a mixed reality HMD252and a core processing component254. The mixed reality HMD252and the core processing component254can communicate via a wireless connection (e.g., a 60 GHz link) as indicated by link256. In other implementations, the mixed reality system250includes a headset only, without an external compute device or includes other wired or wireless connections between the mixed reality HMD252and the core processing component254. The mixed reality HMD252includes a pass-through display258and a frame260. The frame260can house various electronic components (not shown) such as light projectors (e.g., LASERs, LEDs, etc.), cameras, eye-tracking sensors, MEMS components, networking components, etc.

The projectors can be coupled to the pass-through display258, e.g., via optical elements, to display media to a user. The optical elements can include one or more waveguide assemblies, reflectors, lenses, mirrors, collimators, gratings, etc., for directing light from the projectors to a user's eye. Image data can be transmitted from the core processing component254via link256to HMD252. Controllers in the HMD252can convert the image data into light pulses from the projectors, which can be transmitted via the optical elements as output light to the user's eye. The output light can mix with light that passes through the display258, allowing the output light to present virtual objects that appear as if they exist in the real world.

Similarly to the HMD200, the HMD system250can also include motion and position tracking units, cameras, light sources, etc., which allow the HMD system250to, e.g., track itself in 3DoF or 6DoF, track portions of the user (e.g., hands, feet, head, or other body parts), map virtual objects to appear as stationary as the HMD252moves, and have virtual objects react to gestures and other real-world objects.

FIG.2Cillustrates controllers270, which, in some implementations, a user can hold in one or both hands to interact with an artificial reality environment presented by the HMD200and/or HMD250. The controllers270can be in communication with the HMDs, either directly or via an external device (e.g., core processing component254). The controllers can have their own IMU units, position sensors, and/or can emit further light points. The HMD200or250, external sensors, or sensors in the controllers can track these controller light points to determine the controller positions and/or orientations (e.g., to track the controllers in 3DoF or 6DoF). The compute units230in the HMD200or the core processing component254can use this tracking, in combination with IMU and position output, to monitor hand positions and motions of the user. The controllers can also include various buttons (e.g., buttons272A-F) and/or joysticks (e.g., joysticks274A-B), which a user can actuate to provide input and interact with objects.

In various implementations, the HMD200or250can also include additional subsystems, such as an eye tracking unit, an audio system, various network components, etc., to monitor indications of user interactions and intentions. For example, in some implementations, instead of or in addition to controllers, one or more cameras included in the HMD200or250, or from external cameras, can monitor the positions and poses of the user's hands to determine gestures and other hand and body motions. As another example, one or more light sources can illuminate either or both of the user's eyes and the HMD200or250can use eye-facing cameras to capture a reflection of this light to determine eye position (e.g., based on set of reflections around the user's cornea), modeling the user's eye and determining a gaze direction.

FIG.3is a block diagram illustrating an overview of an environment300in which some implementations of the disclosed technology can operate. Environment300can include one or more client computing devices305A-D, examples of which can include computing system100. In some implementations, some of the client computing devices (e.g., client computing device305B) can be the HMD200or the HMD system250. Client computing devices305can operate in a networked environment using logical connections through network330to one or more remote computers, such as a server computing device.

In some implementations, server310can be an edge server which receives client requests and coordinates fulfillment of those requests through other servers, such as servers320A-C. Server computing devices310and320can comprise computing systems, such as computing system100. Though each server computing device310and320is displayed logically as a single server, server computing devices can each be a distributed computing environment encompassing multiple computing devices located at the same or at geographically disparate physical locations.

Client computing devices305and server computing devices310and320can each act as a server or client to other server/client device(s). Server310can connect to a database315. Servers320A-C can each connect to a corresponding database325A-C. As discussed above, each server310or320can correspond to a group of servers, and each of these servers can share a database or can have their own database. Though databases315and325are displayed logically as single units, databases315and325can each be a distributed computing environment encompassing multiple computing devices, can be located within their corresponding server, or can be located at the same or at geographically disparate physical locations.

Network330can be a local area network (LAN), a wide area network (WAN), a mesh network, a hybrid network, or other wired or wireless networks. Network330may be the Internet or some other public or private network. Client computing devices305can be connected to network330through a network interface, such as by wired or wireless communication. While the connections between server310and servers320are shown as separate connections, these connections can be any kind of local, wide area, wired, or wireless network, including network330or a separate public or private network.

FIG.4is a block diagram illustrating components400which, in some implementations, can be used in a system employing the disclosed technology. Components400can be included in one device of computing system100or can be distributed across multiple of the devices of computing system100. The components400include hardware410, mediator420, and specialized components430. As discussed above, a system implementing the disclosed technology can use various hardware including processing units412, working memory414, input and output devices416(e.g., cameras, displays, IMU units, network connections, etc.), and storage memory418. In various implementations, storage memory418can be one or more of: local devices, interfaces to remote storage devices, or combinations thereof. For example, storage memory418can be one or more hard drives or flash drives accessible through a system bus or can be a cloud storage provider (such as in storage315or325) or other network storage accessible via one or more communications networks. In various implementations, components400can be implemented in a client computing device such as client computing devices305or on a server computing device, such as server computing device310or320.

Mediator420can include components which mediate resources between hardware410and specialized components430. For example, mediator420can include an operating system, services, drivers, a basic input output system (BIOS), controller circuits, or other hardware or software systems.

Specialized components430can include software or hardware configured to perform operations for implementing a holographic call between an artificial reality device and mobile device using front and back facing cameras of the mobile device. Specialized components430can include an image capture module434, a spatial relationship analyzer436, an affordance module438, a holographic generation module440, and components and APIs which can be used for providing user interfaces, transferring data, and controlling the specialized components, such as interfaces432. In some implementations, components400can be in a computing system that is distributed across multiple computing devices or can be an interface to a server-based application executing one or more of specialized components430. Although depicted as separate components, specialized components430may be logical or other nonphysical differentiations of functions and/or may be submodules or code-blocks of one or more applications.

The image capture module434can capture, via a mobile device, images of a sending user from both a front-facing camera and a back-facing camera, thus capturing images of the sending user's face and hand (the one not holding the mobile device).

Additional details on using front and back facing cameras of a mobile device to capture views of different body parts of the sending user are provided below in relation to block502ofFIG.5and in relation toFIGS.6and7.

The spatial relationship analyzer436can analyze images and/or depth data captured by a mobile device to determine a first spatial relationship including a distance and orientation between the mobile device and a sending user's face and a second spatial relationship including a distance and orientation between the mobile device and a sending user's hand. Additional details on determining spatial relationships between a mobile device and parts of a sending user are provided below in relation to block504ofFIG.5and in relation toFIG.7.

The affordance module438can determine, based on the spatial relationships determined by spatial relationship analyzer436, whether the mobile device capturing the images is in a capture zone (e.g., is within a threshold distance of the midpoint between the sending user's hand and face, is a threshold distance away from each, and/or is angled to capture each). When the mobile device is not in the capture zone, the affordance module438can provide affordances (e.g., arrows, blurring effects, text, etc.) instructing the sending user on moving the mobile device and/or her hand to have the mobile device in the capture zone. Additional details on determining whether a mobile device is in a capture zone and, if not, providing affordances are provided below in relation to blocks506and508ofFIG.5and in relation toFIGS.7and8.

The holographic generation module440can use the images captured by image capture module434and the spatial relationships determined by spatial relationship analyzer436to generate a representation of the sending user's face and a representation of the sending user's hand, and place them relative to one another by a recipient artificial reality device. In some cases, the representations are part of an avatar of the sending user positioned according to the determined spatial relationships. In other cases, the representations are holograms of the sending user generated and positioned according to the images and determined spatial relationships. Additional details on generating and displaying representations of the sending user are provided below in relation to block510ofFIG.5and in relation toFIGS.9A and9B.

FIG.5is a flow diagram illustrating a process500used in some implementations of the present technology for a mobile device to capture front and back camera images for a holographic call with an artificial reality device. In some cases, process500can be performed in response to a user initiating a holographic call with a mobile device. In various implementations, process500can be performed on a mobile device engaged in a holographic call or on a server system providing services to such a mobile device.

At block502, process500can capture images with both front and back facing cameras on a mobile device. Depending on the capabilities of the mobile device, the mobile device may capture these images simultaneously or may capture the images iteratively first from the camera(s) facing one direction then from the camera(s) facing the other direction. In some implementations, the mobile device may capture RGB, grayscale, and/or depth images. In some implementations, a mobile device may include multiple cameras facing in one or both directions. For example, the mobile device may include an array of cameras allowing the camera to capture images from different perspectives to generate depth data. In other implementations, the cameras may otherwise enable depth capturing, e.g., using a stereoscopic (multi-lens) depth camera; radar, lidar, sonar, or other time-of-flight (ToF) sensors; a structured light system (e.g., where a grid of captured inferred (IR) points are analyzed for distortion or time of flight readings to precisely identify the distance from the IR source to each point); or other depth sensing technologies. In some implementations, the mobile device can obtain just RGB or greyscale images and these images can be applied to a machine learning model trained to estimate depth data for portions of the image. Additional information on such depth estimations is provided in U.S. patent application Ser. No. 17/360,693, filed Jun. 28, 2021, titled “Holographic Calling for Artificial Reality,” which is hereby incorporated by reference in its entirety.

At block504, process500can determine a spatial relationship between the mobile device and a user's face and a spatial relationship between the mobile device and a user's hand. These spatial relationships can be determined by first applying, to each image, a machine learning model (or other computer vision technique) trained to recognize parts of a user (i.e., the user's face and a hand). In some implementations where depth data was determined at block502, the depth data can then be taken for the identified parts of the image to determine the distance to that body part. In other implementations where there is no depth data, the distance can be estimated such as by determining an expected size of the body part and estimating a distance based on the measured size in the image or by computing a distance to a wearable device on the user's hand and/or face (e.g., a smart bracelet or ring on her hand, and/or glasses or earphone on her face and distances between the mobile device and these wearable devices can be measured based on signal travel time measurements). In some implementations, the determined spatial relationships can also determine orientations of the mobile device in relation to the body parts, e.g., whether the mobile device's cameras are pointed at the identified body part or angled to capture an unfocused view of the body part. In some implementations, the spatial relationships determined at block504can also include simultaneous location and mapping (SLAM) measurements for the mobile device, positioning the mobile device within a room. Using one of the mobile device's camera's (e.g., the rear facing camera), the mobile device can determine its position and orientation relative to the world around it. This can act as a root position for the mobile user's face and hand. Thus, the spatial relationship of the mobile user and the user of the artificial reality device can change as the mobile device's position and orientation changes relative to the world around it.

In some implementations, the spatial relationships determined at block504are 1) a distance and direction measurement between the mobile device and the user's face and upper torso and 2) a distance and direction measurement between the mobile device and the user's hand. In other implementations, the measured depth data for the identified body parts can be mapped to portions of a kinematic model (also sometimes referred to as a body model). A kinematic model can specify a body configuration of the sending user, e.g., distances between body points, such as the distance between the wrist and elbow joints, and angles between body parts, such as the angle between the forearm and upper arm or the direction of the head in relation to the shoulders—thus the kinematic model can limit the distances for body parts estimated by process500to those that match how a user's body can actually move. In some cases, a kinematic model can be for a portion of a user, such as just the user's hand and forearm or just the user's face and head. In various implementations, the kinematic model can be specific to the user (e.g., based on measurements of the user), can be estimated for users with a set of characteristics (e.g., based on a user's age, height, gender, weight, etc.), or can be generic to users generally. Additional information on kinematic models is provided in U.S. patent application Ser. No. 17/360,693, filed Jun. 28, 2021, titled “Holographic Calling for Artificial Reality,” which is hereby incorporated by reference in its entirety. Mapping the depth data to the kinematic model can snap the measurements to an actual body configuration, providing more accurate spatial determinations.

At block506, process500can determine whether the spatial relationships determined at block504indicate the mobile device is in a capture zone for the user's hand and face. A capture zone can be defined as a set of spatial properties of the mobile device, including one or more of: the mobile device being a certain proportional measurement between the user's hand and face (e.g., within 15% of the midpoint between the two), the mobile device having a minimum distance to each of the user's hand and the user's face (e.g., at least six inches from each), and/or the mobile device having a direction such that its cameras' centers of focus are each within a threshold distance of the user's face or hand (e.g., the center of camera focus is within 15% of the user's hand or face). In some implementations, only some of these spatial relationships are used for defining the capture zone. If the spatial relationships indicate the mobile device is in the capture zone, process500can continue to block510; otherwise process500can continue to block508.

At block508, process500can provide positioning affordances to the user, directing the user to move the mobile device and/or her hand such that the mobile device is in the capture zone. The mobile device can be displaying a passthrough of the camera facing toward the user's hand with an overlay representation of the other user participating in the holographic call (referred to above as the receiving user). This representation can show the other user as a screen-locked or world-locked virtual object. The affordances can, in various implementations, include other overlays or effects such as a virtual object representing the mobile device (e.g., as a silhouette) located within the capture zone showing how to move or rotate the mobile device, one or more arrows or other direction indicators showing how to move or rotate the mobile device, a blurring effect which makes the view less blurred as the user moves the mobile device closer to the capture zone, words or an audio output instructing the user how to move or rotate the mobile device to be in the capture zone, etc. In one instance the affordance can show a virtual object as if the user is looking through a pipe and causing the user to position and rotate the mobile device so the pipe is focused on the recipient user and positioned in the capture zone. Following block508, process500can repeat as the mobile device is repositioned and additional images are captured.

At block510, process500can cause representations of the sending user's face and hand to be displayed, relative to one another, by an artificial reality device. In various implementations, the representations of the sending user can be generated, from the images captured at block502, on the mobile device, on a computing system intermediate on a network between the mobile device and a recipient artificial reality device, or on the artificial reality device. In some implementations, one or both representations of the sending user can be generated by simply applying the position data determined at block504to an avatar of the sending user (which may be a lifelike model or a synthetic avatar, such as one with cartoon-like features). For example, a pre-scan of the sending user can have been performed creating a model of the sending user with proportions matching the sending user and can have textures applied from images of the sending user. That model can then be provided to the recipient artificial reality device and positioned according to the determined spatial relationships. An example of such a model is provided inFIG.9B.

In some cases, generating the sending user representations can include generating real-time holograms of one or both of the sending user's face (and possibly portions of her upper torso) or hand (and possibly forearm). For example, the depth information from block502can be used to generate 3D meshes of the sending user's face and/or of the sending user's hand and the image RGB data (masked to determine the portions of the images depicting the user's face or hand) can then be applied as a texture onto the generated 3D mesh to create a holographic representation of the portion of the sending user. The receiving artificial reality device can position each of the holographic representations of the user's face and hand relative to each other based on the spatial relationships determined at block504(i.e., such that the user's hand and face are positioned as they are on the sending user). In some implementations, these representations can be filled in to show uncaptured portions of the sending user (e.g., with estimated color and kinematic model data) or these models can show just the captured portions of the sending user (e.g., fading out at the edges as shown inFIG.9A). In some cases, creating the sending user representations can include extrapolating portions of the sending user not shown in the captured images. For example, a sending user may be holding up her hand with her palm facing away from the sending user. The recipient user may be viewing the hologram of the sending user's hand from the opposite side as the mobile device (i.e., the mobile device captures the back of the user's hand, but the recipient user sees the front of the sending user's palm). Thus, the process can include generating a hand hologram that positions the hand according to the determined hand position of the sending user, but paints a generic palm texture onto the portion of the hand that the mobile device did not capture. An example of such a model is provided inFIG.9A. Additional information on generating a hologram of a sending user from image data is provided in U.S. patent application Ser. No. 17/360,693, filed Jun. 28, 2021, titled “Holographic Calling for Artificial Reality,” which is hereby incorporated by reference in its entirety.

In some implementations, while process500is causing the representations of the sending user's face and hand to be displayed by a recipient device, the recipient device is also sending a depiction of the recipient user to the mobile device, which the mobile device can display on its screen. For example, the mobile device may be providing a feed of the camera capturing the sending user's hand and can include the recipient user representation in the feed, e.g., as a world-locked or screen locked virtual object. In some cases, the position of the user's hand in the display can be used to cause interaction with displayed virtual objects and/or with a representation of the recipient user. For example, a shared virtual space can be defined between the sending and receiving users. Objects in this shared virtual space can be displayed on the mobile device's screen and by the recipient user's artificial reality device. An interaction with a virtual object by the sending user can cause a corresponding action with the virtual object as viewed by the recipient user. As another example, the sending user can move her hand to interact with recipient user's hologram (as shown in the overlay on her mobile device screen) such as to perform a high five, hand shake etc. Conversely, the recipient user can interact with sending user's hologram (as shown by her artificial reality device) to perform similar inter-user interactions. When users perform such interactions, both users could receive haptic feedback. For example, the mobile device can vibrate and the artificial reality device can provide haptic feedback via a wrist band or other haptic wearable device. In some cases, the position of the sending user, as displayed by the recipient user's artificial reality device and controlling how the sending user's actions are interpreted in relation to virtual objects in the shared virtual space, can be updated based on the SLAM data captured at block502and corresponding root position for the mobile user's face and hand determined at block504. Process500can repeat as the call between the sending and receiving user continues, and can end when the call ends.

FIG.6is a conceptual diagram illustrating an example600of a user using a mobile device to capture both hand and facial images from a perspective view. In example600, a user602has entered a holographic call with a recipient user who has an artificial reality device. To provide a holographic representation of herself, the user602has positioned her mobile device604between her face606and her hand608. This allows the front camera on her mobile device604to capture images of her face606and the back camera on her mobile device604to capture images of her hand608. With these image, the holographic calling system can generate a holographic representation of the user's face606and a holographic representation of the user's hand608, which the artificial reality device of the recipient user can render in spatial relationship to one another according to the determined positions of the mobile device604between the user's face606and hand608(illustrated further in example700).

FIG.7is a conceptual diagram illustrating a top view of an example700of the user602from example600using the mobile device604to capture both hand and facial images. Example700illustrates the spatial determinations relating the mobile device604with positions of the user602's body parts. In particular, the holographic calling system determines a distance and orientation706between the user's face606and the mobile device604and a distance and orientation708between the user's hand608and the mobile device604. When the holographic calling system causes the artificial reality device of the recipient to render the hand and face holograms of the sending user602, these holograms are positioned relative to one another according to the distance and orientations706and708. Example700also illustrates a capture zone710, which the holographic calling system uses to determine whether the mobile device604is positioned to capture images sufficient to generate hand and face holograms.

FIG.8is a conceptual diagram illustrating an example800of affordances provided via a mobile device for instructing a user on capturing both hand and facial images. In example800, a sending user is using her mobile device604to conduct a holographic call with a recipient user, a representation804of whom is displayed on the mobile device604. The mobile device604is also showing a feed from the back-facing camera, which depicts the sending user's hand608. During the holographic call, the sending user has positioned the mobile device604outside the capture zone710. This caused the holographic calling system to determine that the mobile device604needs to be moved to the right to put it back in the capture zone710, and thus has provided arrow affordance802on the mobile device604's screen, instructing the user to move the mobile device604to the right.

FIGS.9A and9Bare conceptual diagrams illustrating examples900and950of holographic representations provided by a recipient artificial reality device as a result of both hand and facial images captured by a sender mobile device. In example900, the images taken by a front-facing camera of a sending user have been used to generate a holographic representation of the sending user's face902. The images taken by a back-facing camera of a sending user have been used to generate a holographic representation of the sending user's hand904. The holographic representation of the sending user's face902was created by using captured depth data of the sending user to generate a 3D mesh of the sending user's face, with a texture added from captured RGB data of the front-facing cameras. The holographic representation of the sending user's hand904was created by applying a machine learning model, to the images from the back-facing cameras, to obtain a kinematic model of the sending user's hand. The kinematic model was used to generate a 3D mesh of the sending user's hand in that position from an existing hand model sized to match the size of the sending user's hand/arm. The 3D mesh of the sending user's hand was then painted to match clothing and skin tone determined for the sending user. Thus, even through the palm side of the sending user was not captured by the sending user's mobile device, the hologram904of the sending user's hand illustrates a representation of the sending user's palm, with her hand positioned as it was captured. In example900, the holograms fade to black at the edges of where the images were captured.

In example950, the images taken of a sending user by both the front-facing camera and the back-facing camera of the sending user's mobile device have been used to determine a kinematic model of the sending user. The kinematic model was created by applying a machine learning model trained to produce a kinematic model (body point positions) based on images from the front and back facing cameras. The kinematic model was used to position a previously created avatar952of the sending user. The previously created avatar952has a default position with a neutral face and her hands at her sides. However, determined head position954, facial expressions, and hand/arm positions956are applied from the produced kinematic model to cause the previously created avatar952to be positioned to match the pose of the sending user.