ADJUSTING POSE OF VIDEO OBJECT IN 3D VIDEO STREAM FROM USER DEVICE BASED ON AUGMENTED REALITY CONTEXT INFORMATION FROM AUGMENTED REALITY DISPLAY DEVICE

An augmented reality. AR, computing server (200) includes a network interface (202), a processor (204), and a memory (206) storing instructions executable by the processor to perform operations. The network interface is configured to receive through a network a three-dimensional (3D) video stream from a user device during a conference session. The operations identify a video object captured in the 3D video stream, and determine a pose of the video object captured in the 3D video stream. The operations obtain AR context information from an AR display device indicating how the video object is to be posed relative to a physical object viewable through a see-through display of the AR display device, and adjust pose of the video object captured in the 3D video stream based on the AR context information. The operations output the video object to the see-through display for display. Related methods and computer program products are disclosed.

TECHNICAL FIELD

The present disclosure relates to rendering augmented reality (AR) environments and associated AR computing servers, such as network server, and AR display devices, and related operations for displaying video objects through AR display devices.

BACKGROUND

Immersive virtual reality (VR) environments have been developed which provide VR environments for on-line conferencing in which computer generated avatars represent locations of human participants in the meetings. Example software products that provide VR environments for on-line conferencing include MeetinVR, Glue, FrameVR, Engage, BigScreen VR, Mozilla Hubs, AltSpace, Rec Room, Spatial, and Immersed. Example user devices that can display VR environments to participants include Oculus Quest VR headset, Oculus Go VR headset, and personal computers and smart phones running various VR applications.

In contrast to VR environments where human participants only see computer generated graphical renderings, human participants using augmented reality (AR) environments see a combination of computer-generated graphical renderings overlaid on a view of the physical real-world through, e.g., see-through display screens. AR environments are also referred to as mixed reality environments because participants see a blended physical and digitally rendered world. Example user devices that can display AR environments include Google Glass, Microsoft HoloLens, Vuzix, and personal computers and smart phones running various AR applications. There is a need to provide on-line conferencing capabilities in an AR environment.

SUMMARY

Some embodiments disclosed herein are directed to an AR computing server that includes a network interface, a processor, and a memory storing instructions executable by the processor to perform operations. The network interface is configured to receive through a network a three-dimensional (3D) video stream from a user device during a conference session. The operations identify a video object captured in the 3D video stream, and determine a pose of the video object captured in the 3D video stream. The operations obtain AR context information from an AR display device indicating how the video object is to be posed relative to a physical object viewable through a see-through display of the AR display device, and adjust pose of the video object captured in the 3D video stream based on the AR context information. The operations output the video object to the see-through display for display.

In some further embodiments the operation to determine the pose of the video object captured in the 3D video stream, includes to determine pose of features of a face captured in the 3D video stream, the operation to adjust pose of the video object captured in the 3D video stream based on the AR context information includes to rotate and/or translate the features of the face captured in the 3D video stream based on comparison of the pose of the features of the face captured in the 3D video stream to the AR context information indication of how the features of the face are to be posed relative to the physical object viewable through the see-through display of the AR display device.

Some other related embodiments are directed to a corresponding method by an AR computing server. The method includes identifying a video object captured in a 3D video stream received from a user device during a conference session, and determining a pose of the video object. The method obtains AR context information from an AR display device indicating how the video object is to be posed relative to a physical object viewable through a see-through display of the AR display device, and adjusts pose of the video object captured in the 3D video stream based on the AR context information. The method outputs the video object to the see-through display for display.

Some other related embodiments are directed to a corresponding computer program product including a non-transitory computer readable medium storing instructions executable by at least one processor of an AR computing server to perform operations. The operations identify a video object captured in a 3D video stream received from a user device during a conference session, and determine a pose of the video object. The operations obtain AR context information from an AR display device indicating how the video object is to be posed relative to a physical object viewable through a see-through display of the AR display device, and adjust pose of the video object captured in the 3D video stream based on the AR context information. The operations output the video object to the see-through display for display.

Some other related embodiments are directed to a corresponding AR computing server configured to identify a video object captured in a 3D video stream received from a user device during a conference session, and determine a pose of the video object. The AR computing server is further configured to obtain AR context information from an AR display device indicating how the video object is to be posed relative to a physical object viewable through a see-through display of the AR display device, and adjust pose of the video object captured in the 3D video stream based on the AR context information. The AR computing server is further configured to output the video object to the see-through display for display.

Some potential advantages of these embodiments is they enable a human participant during a conference to view through a see-through display of an AR display device a video object, such as another participant, which is being displayed with a pose that is determined based on AR context information. The AR computing server can use various characteristics of AR context information to determine how to pose and scale an image of the video object, such as where to pose a video image of the other participant within a room.

Other AR computing servers, methods, and computer program products according to embodiments will be or become apparent to one with skill in the art upon review of the following drawings and detailed description. It is intended that all such additional AR computing servers, methods, and computer program products be included within this description and protected by the accompanying claims.

DETAILED DESCRIPTION

Embodiments of the present disclosure are directed to providing on-line conferencing capabilities in an AR environment. The AR environment can enable a local participant in a conference to visually experience an immersive presence of a remote participant who's video image is posed relative to real-world physical objects that the local participant views through a see-through display of an AR display device (e.g., AR glasses worn by the local participant).

FIGS.1A-1Cillustrate a sequence of scenarios in which a local participant of an on-line conference is viewing a remote participant through an AR display device and in which a video image of the remote participant is posed based on AR context information in accordance with some embodiments of the present disclosure.

Referring toFIG.1A, a local participant100during an on-line conference session is wearing an AR display device220, illustrated as AR glasses or other AR headset, and views a video image110aof a remote participant of the on-line conference session which is generated by an AR computing server200(shown inFIG.2). The video image110aof the remote participant is displayed through the AR display device220with a pose that is adjusted by the AR computing server200based on AR context information obtained by the AR display device220. As will be explained in further detail below, the AR context information may indicate a real-world physical object which is viewed by the local participant100and, relative to which, the video image110aof the remote participant is to be posed (e.g., rotated, scaled, and/or anchored). InFIG.1A, the AR context information may indicate that the video image110aof the remote participant is to be posed relative to a bed or other furniture in the room. The video image110amay be anchored by the AR computing server200relative to the bed or other furniture, so that when the local participant's100view becomes rotated toward the bed or furniture the video image110abecomes displayed with a pose that is superimposed on the real-world, such as being posed resting on the bed or furniture. The AR context information may select one of a plurality of real-world physical objects (e.g., the bed inFIG.1A) which are captured in a video stream from a camera of the AR display device220. The selected one of a plurality of real-world physical objects is associated by the AR computing server200with the video image110aof the remote participant. The video image110aof the remote participant is then posed (e.g., by adjusting location and angular orientation of the displayed video image110aof the remote participant's head) and scaled in size (e.g., by adjusting size of the displayed video image110aof the remote participant's head), with operations by the AR computing server200so that when displayed on the see-through display of the AR display device220the video image110aof the remote participant's head appears to the local participant100to be naturally posed relative to the selected real-world physical object as-if the remote participant were physically present at that location.

Referring toFIG.1B, the local participant100has moved closer to the physical object (e.g., the bed or adjacent seat) where the video image110aof the remote participant is posed and has changed his direction of view toward the video image of the remote participant. Accordingly, the AR context information is responsibly updated by the AR display device220to indicate the distance and relative poses between the local participant100and the video image of the remote participant. The operations by the AR computing server200respond to the updated context information by adjusting the pose (e.g., adjust location and angular orientation of the displayed remote participant's head) and scaling the size (e.g., adjust size of the displayed remote participant's head) so that when displayed through the see-through display of the AR display device220the adjusted video image110bof the remote participant appears to the local participant100to be naturally posed relative to the selected real-world physical object as-if the remote participant were physically present at that location.

Referring toFIG.1C, the local participant100has moved to a different room and the updated AR context information indicates a location in that room where the remote participant is to be posed. The AR context information may be generated by the AR display device220, e.g., by tracking its movement and pose using motion sensors (such as accelerometers), and/or may be generated by the AR context server200such as by tracking movement of the AR display device220relative to real-world physical objects based on a video stream from a camera of the AR display device220. The location may be designated by the local participant100, such as by selecting the location while being viewed through the AR display device220, and/or the location may be programmatically selected such as will be explained in further detail below. The operations by the AR context server200respond to the updated context information by adjusting the pose (e.g., adjust location and angular orientation of the displayed remote participant's head) and scaling the size (e.g., adjust size of the displayed remote participant's head) so that when displayed through the see-through display of the AR display device220the adjusted video image110cof the remote participant appears to the local participant100to be naturally posed relative to the selected real-world physical object as-if the remote participant were physically present at that location, which is illustrated as being adjacent to a table in a kitchen.

FIG.2illustrates an AR system that includes a user device210which provides a 3D video stream of a user, such as the remote participant referenced inFIGS.1A-1C, to the AR computing server200. The AR computing server200poses an image of the user for display through the AR display device220in accordance with some embodiments of the present disclosure.FIG.3illustrates a combined data flow diagram and flowchart of operations performed by the user device210, the AR computing server200, and the AR display device220in accordance with some embodiments of the present disclosure.

Referring toFIGS.2and3, the user device210uses a 3D camera212to generate300a 3D video stream during a conference session. The user device210may include, but is not limited to, a mobile phone, laptop computer, tablet computer, desktop computer, stand-alone network camera, etc. The 3D camera212may include, but is not limited to, a pair of stereo cameras, a Lidar sensor which maps distance to points on an object using a laser and measuring the time for the reflected light to return to a receiver, or another 3D camera device. The 3D video stream may include a pair of video streams from stereo cameras and/or may include processed information from stereo cameras or a Lidar sensor. Such processed information may include point clouds (e.g., collection of points that represent a 3D shape or feature), meshes (e.g., polygon meshes, triangular meshes, or other shaped meshes converted from point clouds), or color and depth information.

The 3D video stream is provided to the AR computing server200for processing via, for example, a radio access network240and networks250(e.g., private networks and/or public networks such as the Internet). The AR computing server200may be an edge computing server, a network computing server, a cloud computing server, etc. which communicates through the networks250with the user device210and the AR display device220.

The AR computing server200includes at least one processor circuit204(referred to herein as “processor”), at least one memory206(referred to herein as “memory”), and at least one network interface202(referred to herein as “network interface”). Although the network interface202is illustrated as a wireless transceiver which communicates with a RAN240, it may additionally or alternatively be a wired network interface, e.g., Ethernet. The processor204may include one or more data processing circuits, such as a general purpose and/or special purpose processor (e.g., microprocessor and/or digital signal processor), which may be collocated or distributed across the networks250. The processor204is operationally connected to these various components. The memory206, described below as a computer readable medium, stores executable instructions208that are executed by the processor204to perform operations.

Operations by the AR computing server200include identifying310a video object captured in the 3D video stream, and determining312a pose of the video object captured in the 3D video stream. The identification of the video object and determination of its pose may correspond to identifying presence and pose of various types of real-world physical objects in the 3D video stream. For example, the determination operation312may identify the pose of the face, body, and/or features of the face and/or body of the remote participant captured in the 3D video stream, such as by identifying pose of the head, eyes, lips, ears, neck, torso, arms, hands, etc. Additionally or alternatively, the determination operation312may identify the pose of furniture objects captured in the 3D video stream, such as a bed, seat, table, floor, etc. in the rooms illustrated inFIGS.1A-1C.

The operations by the AR computing server200further include obtaining314AR context information from the AR display device220indicating how the video object is to be posed relative to a physical object viewable through a see-through display234of the AR display device220. The operations adjust316pose of the video object captured in the 3D video stream based on the AR context information, and output318the video object to the see-through display234of the AR display device220for display. The AR display device220is configured to render322the video object at a location on the see-through display234which is determined based on the adjusted pose (operation316).

In one embodiment, the AR context information obtained from the AR display device220can indicate, for example, pose of a chair, table, floor, etc. on which the video object (e.g., video image of the remote participant inFIGS.1A-1C) is to be posed through the see-through display234. The AR context information can indicate a pose of the physical object, and the operation by the AR computing server200to adjust316pose of the video object captured in the 3D video stream based on the AR context information can include to adjust pose of the video object captured in the 3D video stream based on comparison of the pose of the video object to the pose of the physical object.

In one embodiment, the AR context information provided320by the AR display device220indicates where a user of the AR display device220has designated that the video object with the adjusted pose is to be displayed. For example, the user may designate a real-world physical object, such as a seat, table, bed, floor, etc., in a room where the video object is to be displayed and anchored relative to the real-world physical object. Referring to the illustrative example ofFIG.1A, the user can designate a physical chair next to the bed where the video image110aof the upper body of the remote participant is to be displayed and anchored. The AR display device220may provide a video stream from a camera232(e.g., 2D or 3D camera) which captures the designated physical chair. The AR computing server200then operates to adjust316the pose of the upper body of the remote participant captured in the 3D video stream based on the pose of the physical chair in the video stream from the camera232and/or based on other AR context information (e.g., input by the user and/or generated by the AR display device220) so that the upper body of the remote participant is viewed by the user through the see-through display234as virtually sitting on the designated physical chair next to the bed.

In one embodiment, the AR context information can be obtained by determining pose of the physical object in a video stream from the camera232of the AR display device220. The operation by the AR computing server200to obtain314the AR context information can include to determine a pose of the see-through display234of the AR display device220relative to the physical object captured in a video stream from a camera232of the AR display device220. The operation to adjust316pose of the video object captured in the 3D video stream can include to adjust pose of the video object captured in the 3D video stream based on comparison of the pose of the video object to the pose of the see-through display234of the AR display device220relative to the physical object captured in a video stream from the camera232of the AR display device220.

In the example ofFIG.2, the AR display device220includes at least one processor circuit224(referred to herein as “processor”), at least one memory226(referred to herein as “memory”), at least one network interface222(referred to herein as “network interface”), and a display device230. The AR display device220may include the camera232which is configured to output a video stream capturing images of what the user (e.g., local participant) is presently viewing. Although the network interface222is illustrated as a wireless transceiver which communicates with a RAN240, it may additionally or alternatively be a wired network interface, e.g., Ethernet. The processor224may include one or more data processing circuits, such as a general purpose and/or special purpose processor (e.g., microprocessor and/or digital signal processor), which may be collocated or distributed across the networks250. The processor224is operationally connected to these various components. The memory226, described below as a computer readable medium, stores executable instructions228that are executed by the processor224to perform operations.

In the illustrated example, the display device230is part of a mobile electronic device236which is releasably held by a head-wearable frame238oriented relative to the see-through display screen234. The display device236is arranged to display information that is projected on the see-through display screen234for reflection directly or indirectly toward the user's eyes, i.e., while wearing the frame238. Although not shown, the frame238may include intervening mirrors that are positioned between the see-through display screen234and the user's eyes and, hence the light may be reflected directly or indirectly toward the user's eyes.

In some other embodiments, the see-through display is part of the display device230which operates to superimpose the adjusted pose video image received from the AR computing server200on a video stream of the real-world captured by the camera232. For example, a user holding the mobile electronic device236can view through the display device230a video stream from the camera232of a room, e.g., including the chair and bed shown inFIG.1A. The processor224can operate to combine (superimpose) the video stream from the camera232with the video object (e.g., the video image110aof the remote participant's body) with the adjusted pose (operation316) received from the AR computing server200. Thus, in the context ofFIG.1A, the user holding the mobile electronic device236can view on the display device230the video stream from the camera232of the room and when the user looks at the physical chair (anchored to the video image110a) the video image110aof the remote participant's body is superimposed on the physical chair. Thus, the see-through display referenced herein may, for example, be a partially reflective screen, such as the display234inFIG.2, or may be a display device on which a video object captured by a remote camera of a user device210is superimposed on a video steam of the real-world captured by a local camera of the AR display device220.

As used herein, the term “pose” refers to the position and/or the orientation of a video object relative to a defined coordinate system (e.g., a video frame from the 3D camera212or the user device210) or may be relative to another device (e.g., the AR display device220). A pose may therefore be defined based on only the multidimensional position of one device relative to another device or to a defined coordinate system, only on the multidimensional orientation of the device relative to another device or to a defined coordinate system, or on a combination of the multidimensional position and the multidimensional orientation.

FIG.4illustrates a flowchart of operations that can be performed by the AR computing server200ofFIGS.2and3in accordance with some embodiments of the present disclosure.

Referring toFIGS.3and4, the operation to adjust316pose of the video object captured in the 3D video stream can include to rotate and/or translate pose400of the video object captured in the 3D video stream (e.g., rotate and/or translate location of the video image110aof the remote participant's body inFIG.1A) based on comparison of the pose of the video object captured in the 3D video stream to the AR context information indication of how the video object is to be posed relative to the physical object viewable through the see-through display234of the AR display device220. In a further embodiment, the operation to adjust316pose of the video object captured in the 3D video stream further includes to scale size402of the video object captured in the 3D video stream based on comparison of a size of the video object captured in the 3D video stream to the AR context information indication of a size of the physical object viewable through the see-through display234of the AR display device220.

In a further example embodiment, the operation to determine312the pose of the video object captured in the 3D video stream, can include to determine pose of features of a face captured in the 3D video stream. In the example ofFIG.1A, the pose of the remote participant's head, eyes, ears, lips, etc. captured in the 3D video stream can be determined312. The operation to adjust316pose of the video object captured in the 3D video stream based on the AR context information can include to rotate and/or translate the features of the face captured in the 3D video stream based on comparison of the pose of the features of the face captured in the 3D video stream to the AR context information indication of how the features of the face are to be posed relative to the physical object viewable through the see-through display234of the AR display device220.

As explained above, the AR context information can be obtained by determining pose of the physical object in a video stream from the camera232of the AR display device220. The AR computing server200may be configured to use a context selection rule to automatically select which physical object among a plurality of physical objects which are captured in a video stream from the camera232of the AR display device220. The operation to determine312(FIG.3) the pose of the see-through display234of the AR display device220relative to the physical object captured in the video stream from the camera232of the AR display device (220), includes to identify poses of a plurality of physical objects captured in the video stream from the camera232of the AR display device220, and to select one of the physical objects from among the plurality of physical objects based on the selected one of the physical objects satisfying a context selection rule. The operation then performs the determination of the pose of the see-through display234relative to the pose of the selected one of the physical objects.

Some illustrative non-limiting examples of context selection rule operations are explained. In one embodiment, the operation by the AR computing server200includes to determine that one of the physical objects captured in the video stream from the camera232of the AR display device220satisfies the context selection rule based on the one of the physical objects having a shape that matches a defined shape of one of: a seat on which the video object captured in the 3D video stream is to be displayed on the see-through display234with a pose viewed as appearing to be supported by the seat; a table on which the video object captured in the 3D video stream is to be displayed on the see-through display234with a pose viewed as appearing to be supported by the table; and a floor on which the video object captured in the 3D video stream is to be displayed on the see-through display234with a pose viewed as appearing to be supported by the floor.

In some further operational embodiments, the AR computing server200operates to adjust color and/or shading of the video object in the video stream from the user device210based on color and/or shading of the real-world physical object being viewed by the user operating the AR display device220in combination with the displayed video object with the adjusted pose. In one embodiment, operation by the AR computing server200includes to adjust color and/or shading of the physical object which is output to the see-through display234for display, based on color and/or shading of the physical object captured in the video stream from the camera232of the AR display device220.

As a local participant moves about a room while viewing through the AR display device220the video image of a remote participant posed relative to a real-world physical object, the relative positioning between the location of the local participant in the virtual location of the posed video image of the remote participant can result in substantial range of adjustments being made to the pose (e.g., rotation and translation) and scaling of size of the remote participant's body being viewed. Some poses may result in the upper torso and head of the remote participant to be viewed through the AR display device220while some other poses may result in only the head or portion of the head being viewed. Moreover, how much of the remote participant's body is captured in the 3D video stream from the user device210may change over time due to, for example, the remote participant moving relative to the camera212of the user device210. To facilitate generation of any desired pose and scaling of the video image of the remote participant, some other operational embodiments of the AR computing server200combine a previously stored image of an extended part (e.g., part of the remote participant's body) of an earlier video object to the video object (e.g., remote participant's head) that is presently captured in the 3D video stream. The extended part may be stored in an image part repository209in the memory206of the AR computing server200as shown inFIG.2. For example, these operations may append the earlier image of a body of the remote participant inFIGS.1A-1Cto the image of the remote participant's face which is presently captured in the 3D video stream.

FIG.5illustrates a flowchart of corresponding operations that may be performed by the AR computing server200in accordance with some embodiments. Referring toFIG.5, the operations extract500an image of an extended part of the video object captured in the 3D video stream at an earlier time during the conference session or from another 3D video stream of another conference session. The extended part of the video object may be extracted by copying to memory only the extended part of the video object without copying other objects, background, etc. in a video frame of the 3D video stream. The extended part of the video object is not captured in the 3D video stream at the time of the determination312of the pose of the video object. An example extended part of a video object can correspond to, for example, a video image of the remote participant's neck, torso, arms, etc. The operations store502the image of the extended part of the video object in the memory for subsequent use. The image of the extended part of the video object may be stored502in the image part repository209of the AR computing server200as shown inFIG.2. The operations adjust504pose of the image of the extended part of the video object retrieved from the memory (e.g., the image part repository209of the AR computing server200shown inFIG.2) and/or pose of the video object captured in the 3D video stream, based on comparison of the pose of the video object captured in the 3D video stream to a pose of the image of the extended part of the video object retrieved from the memory (e.g., the image part repository209). The operations scale506size of the image of the extended part of the video object retrieved from the memory (e.g., the image part repository209) and/or size of the video object captured in the 3D video stream, based on comparison of a size of the video object captured in the 3D video stream to a size of the image of the extended part of the video object retrieved from the memory. The operations then combine508the image of the extended part of the video object with the video object captured in the 3D video stream, to generate a combined video object which is output318to the see-through display234of the AR display device220for display.

The AR computing server200may extract the video object captured in the 3D video stream from the user device210to generate an extracted video stream which is output to the AR display device220for display through the see-through display234. In an illustrative embodiment, the video object is one of a plurality of components of a scene captured in the 3D video stream by the 3D camera212of the user device210. The operation by the AR computing server200to adjust316pose of the video object captured in the 3D video stream includes to extract the video object from the 3D video stream without the other components of the scene. The operation the operation by the AR computing server200to output318the video object to the see-through display234for display includes to output the extracted video object with the adjusted pose.

Although the AR computing server200is illustrated inFIG.2and elsewhere as being separate from the AR display device220, in some other embodiments the AR computing server200is implemented as a component of the AR display device220and/or in another computing device. For example, some of the operations described herein as being performed by the AR computing server200may alternatively or additionally be performed by the AR display device220, the user device210, and/or another computing device.

Further Definitions and Embodiments

Many variations and modifications can be made to the embodiments without substantially departing from the principles of the present inventive concepts. All such variations and modifications are intended to be included herein within the scope of present inventive concepts. Accordingly, the above disclosed subject matter is to be considered illustrative, and not restrictive, and the appended examples of embodiments are intended to cover all such modifications, enhancements, and other embodiments, which fall within the spirit and scope of present inventive concepts. Thus, to the maximum extent allowed by law, the scope of present inventive concepts is to be determined by the broadest permissible interpretation of the present disclosure including the following examples of embodiments and their equivalents, and shall not be restricted or limited by the foregoing detailed description.