Remote user field of view-based camera orienting

In a system including a processor and a computer-readable medium in communication with the processor, the computer-readable medium includes executable instructions that, when executed by the processor, cause the processor to control the system to perform receiving, from a remote system via a communication network, a first remote field of view (FOV) of a remote subject; causing, based on the received first remote FOV, a camera orienting unit to orient a camera in a first orientation, wherein the camera oriented in the first orientation has a first local FOV corresponding to the first remote FOV received from the remote system; upon orienting the camera in the first orientation, causing an image capturing unit to capture a first local image through a display; and transmitting, to the remote system via the communication network, the captured first local image.

BACKGROUND

Video conferencing technologies have become increasingly commonplace. Such technologies are now being used worldwide for a wide variety of both personal and business communications. For example, during a teleconference or other video conferencing session, individuals may “interact” and engage in face-to-face conversations through images and sound captured by digital cameras and transmitted to participants. In an attempt to provide more engaging video conferencing experiences, a set of technologies called “telepresence” have been introduced, which aim to allow participants at different geographical locations to feel as if they were present at the same location. The telepresence has provided certain improvements over conventional video conferencing schemes, including immersion experiences to videoconferencing participants located in different geographical locations. By providing sufficient immersion experiences, the participants may feel the same level of trust and empathy as being face-to-face to each other at the same location. However, in order to provide sufficient immersion experiences, it is necessary to simultaneously capture a number of images of a local scene from a number of perspectives and transmit the captured images to a remote site. Hence, a large number of high speed and high performance cameras (e.g., light field cameras) may need to be operated simultaneously, which results in generating a large amount of data that needs to be encoded and transmitted to a remote site via a communication network. As such, there still remain significant areas for more efficient implementations for telepresence techniques that provide sufficient immersion experiences.

SUMMARY

In an implementation, a system for capturing a local image for transmission to a remote system, includes a display having a front surface facing a local subject and a rear surface facing opposite to the front surface; an image capturing unit including (i) a camera positioned on the rear surface of the display and configured to capture a local image through the display, and (ii) a camera orienting unit configured to orient the camera; a processor; and a computer-readable medium in communication with the processor. The computer-readable medium includes executable instructions that, when executed by the processor, cause the processor to control the system to perform functions of receiving, from a remote system via a communication network, a first remote field of view (FOV) of a remote subject; causing, based on the received first remote FOV, the camera orienting unit to orient the camera in a first orientation, wherein the camera oriented in the first orientation has a first local FOV corresponding to the first remote FOV received from the remote system; upon orienting the camera in the first orientation, causing the image capturing unit to capture a first local image through the display; and transmitting, to the remote system via the communication network, the captured first local image.

In another implementation, a non-transitory computer-readable medium stores executable instructions that, when executed by a processor, cause the processor to control a system to perform receiving, from a remote system via a communication network, a first remote field of view (FOV) of a remote subject; orienting, based on the received first remote FOV, a camera in a first orientation, the camera positioned on a rear surface of a display, wherein the camera oriented in the first orientation has a first local FOV corresponding to the first remote FOV received from the remote system; upon orienting the camera in the first orientation, causing the camera to capture a local image through the display; and transmitting, to the remote system via the communication network, the captured first local image.

In another implementation, a method of operating a system for capturing a local image for transmission to a remote system, the system including (i) a display having a front surface facing a local subject and a rear surface facing opposite to the front surface, and (ii) a camera positioned on the rear surface of the display and configured to capture a local image through the display, includes receiving, from a remote system via a communication network, a first remote field of view (FOV) of a remote subject; orienting, based on the received first remote FOV, the camera in a first orientation, wherein the camera oriented in the first orientation has a first local FOV corresponding to the first remote FOV received from the remote system; upon orienting the camera in the first orientation, causing the camera to capture a first local image through the display; and transmitting, to the remote system via the communication network, the captured first local image.

In another implementation, a system for capturing a local image for transmission to a remote system includes a processor and a computer-readable medium in communication with the processor. The computer-readable medium includes executable instructions that, when executed by the processor, cause the processor to control the system to perform functions of receiving, from a remote system via a communication network, a remote field of view (FOV) of a remote subject; orienting, based on the received remote FOV, the camera in a first orientation, wherein the camera oriented in the first orientation has a local FOV corresponding to the remote FOV received from the remote system; detecting at least one of a position, head-facing direction and eye-gazing direction of the local subject with respect to the local FOV; determining, based on at least one of the detected position, head-facing direction and eye-gazing direction of the local subject with respect to the local FOV, that the local subject is not positioned at a center of the local FOV or the local subject is not gazing at the center of the local FOV; orienting the camera in a second orientation that offsets at least one of a distance between the center of the first local FOV and the detected position of the local subject and an angular difference between the center of the first local FOV and the detected head-facing direction or eye-gazing direction of the local subject; causing the camera to capture a local image with the second orientation; and transmitting, to the remote system via the communication network, the captured first local image.

DETAILED DESCRIPTION

This description is directed to capturing local images of a local site with a camera oriented according to a remote subject's field of view (FOV). Upon receiving the remote subject's FOV, the camera may be oriented by rotating (e.g., panning, tilting, diagonally shifting, etc.) the camera, controlling zooming of the camera and/or moving the camera on a two-dimensional (2D) plane or in a three-dimensional (3D) space to have a local FOV corresponding to the remote subject's FOV. Then, a local image is captured with the camera and transmitted to the remote system. When the remote subject moves and the remote subject's FOV is changed, an updated remote FOV is received from the remote system. The camera is then re-oriented to have an updated FOV which corresponds to the updated remote subject's FOV, and a new local image is captured and transmitted to the remote system. Based on the local images, the remote system may display a view of the local site that changes in response to the movements of the remote subject, which may provide immersive videoconferencing/telepresence experience to the remote subject. By orienting the camera to have a FOV that corresponds to the remote subject's FOV, only one camera (or a few more) may be needed and only the images captured by one camera may need to be processed and encoded for transmission in order to provide images of the local site that is actively responsive to the remote subject's movement. Hence, this disclosure provides technical solutions to the technical problems with other image capturing approaches that require a large number of cameras that simultaneously capture the same scene or object from different perspectives and a high processing power to filter all the images captured by the cameras and encode filtered images for transmission. Also, the amount of data for transmitting the captured images of the local site may be significantly reduced, thereby eliminating a need for a higher transfer rate network connection, which is not readily available or too expense for most consumers. Also, it is not necessary for users to use or wear any additional pieces of equipment, such as, an augmented reality (AR) or virtual reality (VR) set, or the like, which could alter the appearance and impression of the users.

This disclosure is not limited to physically rotating, moving or zooming the camera as similar or same results may be achieved by processing a captured image to emulate rotating, moving or zooming of the camera. Thus, a physical approach (i.e., physically rotating, moving and zooming a camera and a software approach (i.e., processing a captured image to emulate rotating, moving and zooming a camera may be used in an isolated or complimentary manner. In addition, an image captured by using the physical or software approach, or both, may be further processed to, for example, normalize the captured image, remove unnecessarily image portions, reduce the amount of data, etc. For example, due to inherent lens characteristics, an image captured by the camera may be distorted (e.g., fisheye lens effect). In order to reduce or eliminate such distortion, the captured image may be modified or warped to stretch or compress different portions of the captured image in an attempt to compensate the distortion, which is commonly known as warping. Hence, the physical and software approaches and image normalization may be discretionally used in an isolated or complimentary manner.

With this overview, attention is now turned to the figures to described various implementations of the presenting teachings.FIG. 1illustrates Alice10and Bob20standing at the same site and looking at each other through a physical window5positioned between them. When a camera (not shown) is used to capture a scene viewed by Alice10through the window5, an ideal location for capturing the scene including Bob20would be Alice's eye location12. Alice10does not have a full view of Bob20because Alice's view of Bob20is limited by the size of the window5, which blocks Bob's low leg portion. Such Alice's view of Bob20and the surrounding, which is limited by the window5, is referred to as Alice's FOV16. As Alice10moves, a view through the window5that is seen by Alice10changes. For example, when Alice10moves towards the window5, Alice's FOV16widens, Alice can see more areas through the window5, which includes Bob's lower leg portion. Also, as Alice10gets closer to Bob20, Bob20occupies a larger portion of Alice's FOV16, making Bob20look bigger. When such natural view changes are captured and displayed on a screen, Alice10may perceive Bob's images as more realistic and may feel the same or similar level of trust and empathy as being face-to-face at the same location.

FIG. 2illustrates Alice10and Bob20located at first and second sites110and120, respectively, that are geographically remote from each other. Alice10and Bob20are using video conferencing or telepresence devices200A and200B (collectively referred to as devices200), respectively, to have a video conference with each other. The devices200may be communicatively linked via a network30, which may be a wired network, a wireless network, or a combination thereof. As will be described in more detail in later examples, Alice's device200A may transmit Alice's FOV16A to Bob's device200B via the network30. Alice's FOV16A may be determined based on Alice's eye location12A. In response to receiving Alice's FOV16A, Bob's device200B may capture an image of the second site120with a FOV26A which corresponds to Alice's FOV16A, and transmit the captured image to Alice device200A. Then, Alice's device200A may display the image of the second site120that is captured based on Alice's FOV16A. When Alice10moves, Alice's device200A may determine Alice's new FOV16B, which may be based on Alice's new eye location12B, and transmit the new FOV16B to Bob's device200B. For example, Alice10may move closer to the device200A, which may widen her FOV. Upon receiving the new FOV16B, Bob's device200B may capture a new image of the second site120with a new FOV26B which corresponds to Alice's new FOV16B. For example, Bob's device200B may capture the second site120with the FOV26B that is wider than the previous FOV26A, which reflects the changes of Alice's FOV from the narrower FOV16A and the wider FOV16B. The captured image of the second site120is then transmitted to Alice's device200A.

In an implementation, Alice's device200A may continuously determine and transmit Alice's FOV to Bob's device200B by, for example, continuously tracking the position of Alice with respect to Alice's device200A. For example, Alice's device200A may include a camera that detects positions of Alice's head, eye sockets, eye-gazing direction, and/or the like. Based on Alice's head position, the device200A may determine Alice's coarse position with respect to her device200A. Then, using a center position of Alice's eye sockets or eyeballs or eye-gazing direction, the device200A may determine a distance between Alice and her device200A, Alice's eye gazing direction and angle with respect to the device200A. Alice's device200A may also estimate Alice's current or future FOVs based on Alice's past FOVs and transmit the estimate current or future FOVs to Bob's device200B. Alternatively or additionally, Bob's device200B may estimate Alice's current or future FOVs based on Alice's past FOVs. Such estimation may be performed by tracking Alice's eye locations, which is described in U.S. Pat. No. 10,554,928, issued on Feb. 4, 2020 and titled “TELEPRESENCE DEVICE” and U.S. patent application Ser. No. 15/955,672, filed on Apr. 17, 2018 titled “TELEPRESENCE DEVICES OPERATION METHODS,” which are incorporated by references in their entirety.

FIG. 3illustrates an exploded view of an implementation of an image capturing system300, which may be included in Alice and Bob's telepresence devices200A and200B. For example, the system300may be included in Bob's device200B. The image capturing system300may include a display310, image capturing unit320, a back panel330and a control unit340. The display310may be a transparent flat panel display panel, such as organic light-emitting diode (OLED) panel, etc., of which a front surface312is facing Bob20and a rear surface314facing opposite to the front surface312. The image capturing unit320may be sandwiched between the display310and back panel330. The back panel330may have dark, opaque or non-reflecting surfaces to avoid reflecting lights and make the image capturing unit320invisible or less visible when viewed from the front surface312of the display310. The back panel330may be part of an enclosure or construction (not shown) attached to the rear surface314of the display310. Such enclosure or construction may include one or more dark, opaque or non-reflecting surfaces. For example, the enclosure or construction may include the back panel330and four side panels that are also dark, opaque or non-reflecting and surrounding the space between the display310and back panel330. The system300may be assembled together and contained within a housing (not shown), which may have an appearance similar to a flat panel display device (e.g., television, monitor, tablet, etc.).

The image capturing unit320may include a camera322positioned on a rear surface314of the display310, and a camera orienting unit324configured to orient the camera322such that the camera322has a FOV corresponding to a FOV received from a remote system, such as Alice's device200A. The camera322may have a wide angle lens and may capture images at a higher resolution (e.g.,4K or8K resolution) than that of an image transmitted to Alice's device200A. This may allow to electronically process the captured image data for emulating panning, titling, diagonally shifting, zooming, etc. without lowering the resolution of the transmitted image. This may temporarily or permanently reduce or eliminate a need for physically rotating, moving or zooming the camera322. The camera orienting unit324may control an orientation of the camera322. For example, the camera322may be fixed at a location corresponding to a center316of the rear surface314of the display310. The camera orienting unit324may be connected to the camera322and orient the camera322to have a desired FOV by tilting, panning and diagonally shifting the camera322. The camera322may be diagonally shifted at any desired angle by combining tilting and panning. The camera orienting unit324may also control zooming of the camera322.

The control unit340may receive Alice's FOV from Alice's device200A via the network30. The control unit340may then cause, based on Alice's FOV, the camera orienting unit324to orient the camera322in a particular orientation such that the camera322may have a FOV corresponding to Alice's FOV. Upon orienting the camera322to have the FOV corresponding to Alice's FOV, the control unit340may cause the image capturing unit320to capture a local image through the display310. The control unit340may process the image captured by the camera322to emulate rotating, moving and zooming of the camera322. Such physical approach (i.e., physically rotating, moving and zooming of the camera322) and software approach (i.e., processing the captured image to emulate rotating, moving and zooming of the camera322) may be discretionally selected in an alternative or complementary manner. For example, when the received FOV requires zooming, the control unit340may decide whether such zooming can be done by processing the capture image. If the required zooming is within an operational range of the software approach, the control unit340may opt to perform the software approach. This may reduce wear and tear of the mechanical parts constituting the image capturing unit320and camera322. In addition to capturing the image via the physical and/or software approaches, the control unit340may perform image normalization of the captured image by, for example, warping (e.g., stretching or compressing different portions of the captured image) to compensate distortions introduced to the captured image. The control unit340may then transmit image data including the captured and compensated local image to Alice's device200A via the network30.

FIGS. 4A, 4B, 4C, 5A and 5Bshow examples of the camera orienting unit324orienting the camera322by, for example, rotating the camera322to tilt and pan the FOV of the camera322.FIG. 4Ashows a plane400which has the substantially same width (on an “X” axis) and height (on a “Y” axis) as the rear surface314of the display310and is substantially parallel to the rear surface314of the display310. The plane400may be flush with or slightly spaced apart from the rear surface314. The camera orienting unit324may tilt up and down the FOV of the camera322with respect to a line410substantially parallel to the X axis of the plane400, which is shown by an arrow412.FIG. 4Bshows the camera322being rotated with respect to the line410in a direction shown by an arrow420to tilt up the FOV of the camera322.FIG. 4Cshows the camera322being rotated with respect to the line410in a direction shown by an arrow430to tilt down the FOV of the camera322. As shown inFIGS. 5A and 5B, the camera orienting unit324may also rotate the camera322with respect to a line500on the plane400that is substantially parallel to the Y axis of the plane400.FIG. 5Ashows the camera322being rotated with respect to the line500in a direction shown by an arrow510to pan the FOV of the camera322to the right.FIG. 5Bshows the camera322being rotated with respect to the line500in a direction shown by an arrow520to pan the FOV of the camera322to the left. The camera orienting unit324may also rotate the camera322in any diagonal directions by, for example, combining of the vertical rotation shown inFIGS. 4A, 4B and 4Cand horizontal rotation shown inFIGS. 5A and 5B. Further, the camera orienting unit324may control zooming of the camera322to adjust a FOV size of the camera322.

In addition to rotating the camera322and controlling zooming of the camera322, the camera orienting unit324may move the camera322to orient the camera322to have a desired FOV that corresponds to Alice's FOV. For example,FIGS. 6A, 6B and 6Cillustrate the camera orienting unit324including a sliding system600configured to move the camera322freely over the rear surface314of the display310. The sliding system600may be motorized to position the camera322at a desired point on a plane defined by X and Y axes and substantially parallel to the rear surface314of the display310. For example, based on Alice's FOV received from Alice's device200A, the control unit340may detects a change of Alice's FOV which requires the camera322to be positioned near the top left corner of the rear surface314of the display310. The control unit340may then calculate coordinates on the X and Y axes of the sliding system600that corresponds to the desired position of the camera322on the rear surface314of the display310. As shown inFIG. 6B, the sliding system600may then move the camera322along the X and Y axes of the plane to position the camera322near the top left corner of the rear surface314of the display310. Similarly, when a new FOV is received from Alice's device200A which requires the camera322is to be positioned near the bottom right corner of the rear surface314of the display310, the sliding system600may then move the camera322along the X and Y axes of the plane to position the camera322near the bottom right corner of the rear surface314of the display310, as shown inFIG. 6C.

The camera orienting unit324may be configured to move the camera322in a 3D space on the rear surface314of the display310. For example, as shown inFIG. 7A, the camera orienting unit324may include a 3D sliding system700, which may include a XY sliding system710and a Z sliding system720. Similar to the sliding system600shown inFIGS. 6A, 6B and 6C, the XY sliding system710may be configured to move the camera322along the X and Y axes of the plane substantially parallel to the rear surface314of the display310. The Z sliding system720may be configured to move the XY sliding system710along a Z axis that is perpendicular to both the X and Y axes of the plane. Hence, the Z sliding system720may control a distance between the camera322and the rear surface314of the display310.FIG. 7Bshows the Z sliding system720moving the XY sliding system710toward the rear surface314of the display310as shown by an arrow702.FIG. 7Cshows the Z sliding system720moving the XY sliding system710away the rear surface314of the display310as shown by an arrow704. As such, the camera orienting unit324may orient the camera322to have a desired FOV by rotating the camera322, controlling zooming of the camera322, moving the camera322on a 2D plane or 3D space over the rear surface314of the display310.

In an implementation, two or more cameras may be used. For example, inFIG. 6A, the plane may be divided into a number of smaller areas of the same or different sizes, such as two areas (e.g., two halves, one large area and one small area, etc.), four arears (e.g., four quarters, two larger areas and two smaller areas, five areas (e.g., one center area and four side areas, etc.), etc. and each area may be provided with a camera which may be controlled, rotated and moved by the camera orienting unit324. Similarly, inFIG. 7A, the 3D space behind the display310may be divided into a number of smaller spaces, and each space may be provided with its own camera which may be controlled, rotated and moved by the camera orienting unit324. By having more camera to cover different areas, an amount of time to move the camera from one position to another may be reduced, and the camera orienting unit324may move and rotate the cameras in a more robust and responsive manner. This may also allow to provide a number of different views to remote viewers.

FIGS. 8A, 8B, 8C, 8D and 8Eare top views of Alice10and Bob20facing their respective devices200A and200B, in which the camera322in Bob's device200B is oriented based on Alice's FOV in order to have a local FOV corresponding to Alice's FOV.FIG. 8Ashows Alice's device200A having a camera80that captures an image of Alice10. The camera80may be an infrared (IR) camera or depth camera, of which the main purpose is capturing Alice's head or eye direction or eye-gazing direction. Alternatively, the camera80may be the camera322of Alice's device200A. Based on the captured image of Alice10, Alice's device200A may detect various position and direction information, such as Alice's position with respect to the display310, Alice's head facing direction, Alice's eye gazing direction, etc. Based on the detected position and direction information, Alice's device200A may determine Alice's FOV16A. A configuration (e.g., width and height) of the display310may also be considered in determining Alice's FOV16A because Alice's FOV16A may be limited to a view defined by a display area of the display310or an application window displaying the transmitted image.

In an implementation, the camera80may continuously or periodically capture images of the first site110. Based on the captured images of the first site110, Alice's device200A may track Alice's movements, positions with respect to the display310, head-facing directions, eye-gazing directions, etc. and continuously or periodically determine and transmit Alice's FOV to Bob's device200B. Referring toFIG. 8A, Alice's FOV16A may be defined by Alice's eye location (e.g., the middle point between Alice's eye sockets)90A with respect to the display310, Alice's eye gazing direction92A with respect to the display310, view boundaries defined by the width and height of the display310, etc. Alice's device200A may detect that Alice10is positioned at a horizontal center with respect to the display310and looking at the center of the display310in a direction perpendicular to the display310. Based on the detected position and direction information, Alice's device200A may determine Alice's FOV16A that is substantially symmetrical with respect to the center of the display310. Alice's device200A may then transmit Alice's FOV16A to Bob's device200B via the network30.

Upon receiving Alice's FOV16A, Bob's device200B may determine a local FOV160A of the camera322with respect to its display310. For example, Bob's device200B may determine that, based on Alice's FOV16A, a virtual eye location190A with respect to the display310corresponding to Alice's eye location90A, a virtual eye gazing direction192A corresponding to Alice's eye gazing direction92A, a virtual FOV boundary defined by edges of the display310, etc. Based on the virtual eye location190A, virtual eye gazing direction192A, virtual FOV boundary, etc., Bob's device200B may perform rotating or moving the camera322, controlling zooming of camera322to orient the camera322, etc. to have the local FOV160A corresponding to Alice's FOV16A. For example, the camera322may be moved to be positioned on a line extending from the virtual eye location190A along the virtual eye gazing direction192A. The camera322may also be rotated to face in a direction that is substantially the same as the virtual eye gazing direction192A. The camera322may be also controlled to zoom in or out or moved closer to or away from the display310based on a distance between the virtual eye location190A and the display310. Upon being oriented to have the local FOV160A corresponding Alice's FOV16A, the camera322may be controlled to capture an image of the second site120. The captured image may then be transmitted to Alice's device200A and displayed on the display310of Alice's device200A.

Due to size and configuration restrictions, the camera322at Bob's device200B may not be positioned to have the same distance from the display310as the virtual eye location190A is virtually positioned. In some situations, the camera322may be positioned closer to the display310than the virtual eye location190is positioned, and hence the local FOV160A may be wider than Alice's FOV16A, capturing areas of the second site120that are outside Alice's FOV16A. To match Alice's FOV16A, Bob's device200A may process the captured image to eliminate data related to the area outside Alice's FOV, which may reduce the amount of data transmitted to Alice's device200A. Bob's device200A may also perform up-scaling or down-scaling of the capture image to a desired image format (e.g., 1920×1080 resolution) for transmission to Alice's device200A. Further, the captured image may be warped to compensate distortions introduced to the captured image. Alternatively, Bob's device200B may transmit the image as captured by the camera322to Alice's device200A, which may then process the received image to eliminate the areas that cannot be displayed on its display310or to perform warping of the received image.

FIG. 8Bshows Alice10moving closer to the display310of her device200A, which may change Alice's eye location from location90A to90B. Upon detecting Alice's movement, Alice's device200A may determine Alice's new FOV16B, which is wider than her previous FOV16A shown inFIG. 8A, and transmit the new FOV16B. Upon receiving Alice's new FOV16B, Bob's device200B may move the camera322closer to the display310and/or physically or electronically control zooming of the camera322to orient the camera322to have a new local FOV160B corresponding to Alice's FOV16B. Bob's device200B may then control the camera322to capture an image of the second site120. Due to the shortened distance between the camera322and Bob20, the captured image may show more peripheral areas of the second site120that were not shown in the previous image captured with the FOV160A shown inFIG. 8A, which may result in Bob's image occupying a smaller portion of the captured image. However, when the captured image is displayed at Alice's device200A, Bob20may look bigger to Alice10because Alice10is physically closer to her display310.

FIG. 8Cshows Alice10moving to the left from her previous position inFIG. 8Awhile maintaining the distance from the display310and keeping her eye-gazing direction perpendicular to the display310. Such movement may result in Alice10having a new FOV16C, which allows Alice10to see more peripheral areas on the right side and less peripheral areas on the left side when compared to the previous FOV16A. Upon receiving the new FOV16C, Bob's device200B may move the camera322to have a new local FOV160C corresponding to Alice's FOV16C. To capture a perspective view of Bob20from the same position as Alice's new eye location90C, the camera322may be moved to be on a line extending perpendicular to the display310from a virtual eye position190C corresponding to the Alice's eye position90C. Also, similar to Alice's head direction, the camera322may face in a direction perpendicular to the display310. Such synchronization between Alice's movement and position and the orientation of the camera322may ensure a natural eye contacts between Alice10and Bob's image displayed via Alice's device200A.

FIG. 8Dshows Alice10positioned on the same position asFIG. 8Cwith her entire body (including her head) tilted in a clockwise direction while maintaining her eye-gazing direction perpendicular to her head. Upon detecting such changes, Alice's device200A may determine a new FOV16D, which may be slightly different from her previous FOV16C. For example, the change to Alice's head-facing direction may result in shifting Alice's eye location from the location90C to a new eye location90D, which is slightly to the right from the previous FOV16C and slight away from the display310. Alice's new FOV16D may allow Alice10to see the left peripheral area slightly more and the right peripheral area slightly less. Upon receiving Alice's new FOV16D, Bob's device200B may orient the camera322by slightly moving the camera322to the right and slight away from the display310and horizontally tilting the camera322in a clockwise direction such that the camera322has a new local FOV160D corresponding to Alice's FOV16D. Then, Bob's device200B may cause the newly oriented camera322to capture an image of the second site120, which is transmitted to Alice's device200A to display the image of the second site120captured with the new FOV160D.

FIG. 8Eshows Alice10positioned on the same position asFIG. 8Dand having the same head-facing direction asFIG. 8Dwith her eye-gazing direction being changed to the left side, which may result in slightly changing her eye location and also slightly changing her FOV from the previous FOV16D to a new FOV16E. Upon receiving the new FOV16E, Bob's device200B may move and rotate the camera322to have a new local FOV160E corresponding to Alice's new FOV16E. Then, Bob's device200B may operate the camera322to capture a new image of the second site120, which may show the right peripheral area slightly more and the left peripheral area slightly less, and transmit the captured image to Alice's device200A, which displays the newly captured image of the second site120.

As such, Bob's device200B may be configured to orient the camera322in response to changes to Alice's FOV. In some circumstances, such changes to Alice's FOV may be very small. Bob's device200B, however, may be configured to and operate to be actively responsive to such small FOV changes to ensure that the images of the second site120change in response to Alice's movements. This may allow Alice10to perceive Bob's images displayed via her device200A much more realistic, thereby providing more immersive and engaging video conferencing experiences. Also, by rotating or moving the camera322, controlling zooming of the camera322, etc. the camera322may be oriented to have a FOV which corresponds to Alice's FOV, and hence only one (or a few more) camera may be needed to achieve a more lifelike video conferencing experience. Hence, Bob's device200B may not need a large number of cameras, such as light field cameras, which are expensive and relatively bulky, to capture images to match Alice's constantly changing FOV. Also, by using a single (or a few more) camera, an amount of data transmitted to a viewer's device is significantly reduced. Further, Alice's device200A may display real life-like images of Bob without using additional enhancement equipment (e.g., AR/VR headgear, glasses, etc.).

FIG. 9is a flow diagram showing an implementation of a process of operating a system for capturing a local image for transmission to a remote system. The system may include (i) a display having a front surface facing a local subject and a rear surface facing opposite to the front surface, and (ii) a camera positioned on the rear surface of the display and configured to capture a local image through the display. At step910, the system may receive, from a remote system via a communication network, a first remote field of view (FOV) of a remote subject. At step920, the system may orient, based on the received first remote FOV, the camera in a first orientation, wherein the camera oriented in the first orientation has a first local FOV corresponding to the first remote FOV received from the remote system. At step930, upon orienting the camera in the first orientation, the system may cause the camera to capture a first local image through the display. At step940, the system may transmit, to the remote system via the communication network, the captured first local image.

In some circumstances, Bob20may be positioned away from the center of the local FOV or Bob's head-facing direction or eye-gazing direction may be off from the center of the local FOV. Bob's device200B may then perform an offsetting operation to have Bob's image to be positioned at the center of the local FOV or Bob's head-facing direction or eye-gazing direction is directed towards the center of the local FOV. For example,FIG. 10shows Alice10being positioned at the same location as shown inFIG. 8Awith the same eye location90A and FOV16A. On the other hand, Bob20is positioned on the right side of the local FOV160A, which is determined based on Alice's FOV16A. In this situation, a captured image of the second site120may show Bob20shifted to the right from the center of the local FOV160A. Upon detecting such deviation, Bob's device200B may adjust the orientation of the camera322to offset a distance between the center of the local FOV160A and Bob's current portion within the local FOV160A. For example, Bob's device200B may re-orient the camera322by horizontally moving the camera322to the right such that the camera322may have a new FOV160X, in which Bob20is positioned at the center. This may allow Bob's device200B to capture and transmit an image of the second site120, in which Bob20is positioned at the center. Hence, Alice10may not need to tilt her body or head to the right to maintain the eye contact with Bob's image. In certain circumstances, Bob20may be positioned further away from the center of the local FOV, which may make it difficult for the camera322to capture Bob's complete image even after adjusting the orientation of the camera322. In such case, Bob's device200B may warp the captured image to make Bob's image look as normal as possible prior to transmitting to Alice's device200A. Such image normalization may be performed by Alice's device200A after receiving the captured image form Bob's device200B. Such image normalization may be performed periodically at a predetermined interval (e.g., 30 frames or 5 minutes) by Bob's device200B, Alice's device200A or both in a complementary manner.

FIG. 11shows Alice10being positioned at the same location as shown inFIG. 8Awith the same eye location90A and FOV16A. Bob20is positioned at the center of the local FOV160A corresponding to Alice's FOV16A with his head-facing direction and eye-gazing directions shifted to a left side of the local FOV16A. This may happen when Alice's image is displayed via a window positioned on the right side of the front surface312of the display310at Bob's device200B. In such situation, Bob20may be making an eye contact with Alice's image displayed on the display310, but Bob's image20captured with the local FOV160A may not be making an eye contact with Alice10. To offset an angular difference between the center of the local FOV160A and Bob's head-facing direction or eye-gazing direction, Bob's device200B may adjust the orientation of the camera322to have a new local FOV160Y such that Bob's position, head-facing direction and/or eye-gazing direction are positioned at the center of the local FOV160Y. The orientation of the camera322may be adjusted by horizontally moving the camera322to the left and rotating the camera322to be on a line1100coinciding with Bob's head-facing direction or eye-gazing direction1000. Due to constructional limitations, the camera322may not be oriented to accurately match Bob's head-facing direction or eye-gazing direction1000. In such cases, the camera322may be moved and rotated as much as possible to match Bob's head-facing direction or eye-gazing direction1100. Bob's device200B may then capture an image of the second site120and warp Bob's image to look as normal as possible. As such, the orientation of camera322may be adjusted to offset a distance between the center of the local FOV and Bob's position with respect to the local FOV. The orientation of camera322may also be adjusted to offset an angular difference between the center of the local FOV and Bob's head-facing direction or eye-gazing direction. This may allow Alice10to maintain an eye contact with Bob's image displayed on Alice's device200A even when Bob is not positioned or looking at the center of the display310of Bob's device200B.

FIG. 12is a flowchart showing another implementation of orienting a camera to offset a distance between the center of the local FOV and a local subject's position with respect to the local FOV or an angular difference between the center of the local FOV and a head-facing direction or eye-gazing direction. Such offsetting may be performed periodically (e.g., once in every 30 frames) or when a detected movement exceeds a predetermined threshold (e.g., 1 inch). At step1210, the system may receive, from a remote system via a communication network, a remote field of view (FOV) of a remote subject. At step1220, the system may orient, based on the received remote FOV, the camera in a first orientation, wherein the camera oriented in the first orientation has a local FOV corresponding to the remote FOV received from the remote system. At step1230, the system may detect at least one of a position, head-facing direction and eye-gazing direction of the local subject with respect to the local FOV. At step1240, the system may determine, based on at least one of the detected position, head-facing direction and eye-gazing direction of the local subject with respect to the local FOV, that the local subject is not positioned at a center of the local FOV or the local subject is not gazing at the center of the local FOV. At step1250, the system may orient the camera in a second orientation that offsets at least one of a distance between the center of the first local FOV and the detected position of the local subject and an angular difference between the center of the first local FOV and the detected head-facing direction or eye-gazing direction of the local subject. At step1260, the system may cause the camera to capture a local image with the second orientation. At step1270, the system may transmit, to the remote system via the communication network, the captured first local image.

FIG. 13is a block diagram showing an example computer system1300upon which aspects of this disclosure may be implemented. The computer system1300may include a bus1302or other communication mechanism for communicating information, and a processor1304coupled with the bus1302for processing information. The computer system1300may also include a main memory1306, such as a random-access memory (RAM) or other dynamic storage device, coupled to the bus1302for storing information and instructions to be executed by the processor1304. The main memory1306may also be used for storing temporary variables or other intermediate information during execution of instructions to be executed by the processor1304. The computer system1300may implement, for example, the telepresence devices200A and200B.

The computer system1300may further include a read only memory (ROM)1308or other static storage device coupled to the bus1302for storing static information and instructions for the processor1304. A storage device1310, such as a flash or other non-volatile memory may be coupled to the bus1302for storing information and instructions.

The computer system1300may be coupled via the bus1302to a display1312, such as a liquid crystal display (LCD), for displaying information. One or more user input devices, such as a user input device1314, cursor control1316, etc. may be coupled to the bus1302, and may be configured for receiving various user inputs, such as user command selections and communicating these to the processor1304, or to the main memory1306. The user input device1314may include physical structure, or virtual implementation, or both, providing user input modes or options, for controlling, for example, a cursor, visible to a user through display1312or through other techniques, and such modes or operations may include, for example virtual mouse, trackball, or cursor direction keys.

The computer system1300may include respective resources of the processor1304executing, in an overlapping or interleaved manner, respective program instructions. Instructions may be read into the main memory1306from another machine-readable medium, such as the storage device1310. In some examples, hard-wired circuitry may be used in place of or in combination with software instructions. The term “machine-readable medium” as used herein refers to any medium that participates in providing data that causes a machine to operate in a specific fashion. Such a medium may take forms, including but not limited to, non-volatile media, volatile media, and transmission media. Non-volatile media may include, for example, optical or magnetic disks, such as storage device1310. Transmission media may include optical paths, or electrical or acoustic signal propagation paths, and may include acoustic or light waves, such as those generated during radio-wave and infra-red data communications, that are capable of carrying instructions detectable by a physical mechanism for input to a machine.

The computer system1300may also include a communication interface1318coupled to the bus1302, for two-way data communication coupling to a network link1320connected to a local network1322. The network link1320may provide data communication through one or more networks to other data devices. For example, the network link1320may provide a connection through the local network1322to a host computer1324or to data equipment operated by an Internet Service Provider (ISP)1326to access through the Internet1328a server1330, for example, to obtain code for an application program.

In the following, further features, characteristics and advantages of the invention will be described by means of items:

Item 1. A system for capturing a local image for transmission to a remote system, comprising: a display having a front surface facing a local subject and a rear surface facing opposite to the front surface; an image capturing unit comprising (i) a camera positioned on the rear surface of the display and configured to capture a local image through the display, and (ii) a camera orienting unit configured to orient the camera; a processor; and a computer-readable medium in communication with the processor. The computer-readable medium comprising executable instructions that, when executed by the processor, cause the processor to control the system to perform functions of: receiving, from a remote system via a communication network, a first remote field of view (FOV) of a remote subject; causing, based on the received first remote FOV, the camera orienting unit to orient the camera in a first orientation, wherein the camera oriented in the first orientation has a first local FOV corresponding to the first remote FOV received from the remote system; upon orienting the camera in the first orientation, causing the image capturing unit to capture a first local image through the display; and transmitting, to the remote system via the communication network, the captured first local image.

Item 2. The system of Item 1, wherein, for causing the camera orienting unit to orient the camera in the first orientation, the instructions, when executed by the processor, further cause the processor to control the system to perform causing the camera orienting unit to rotate the camera or control zooming of the camera.

Item 3. The system of Item 2, wherein, for causing the camera orienting unit to orient the camera in the first orientation, the instructions, when executed by the processor, further cause the processor to control the system to perform causing the camera orienting unit to move the camera on the rear surface of the display.

Item 4. The system of Item 3, wherein, for causing the camera orienting unit to orient the camera in the first orientation, the instructions, when executed by the processor, further cause the processor to control the system to perform at least one of: causing the camera orienting unit to move the camera on a plane substantially parallel to the rear surface of the display; and causing the camera orienting unit to move the camera in a three-dimensional (3D) space on the rear surface of the display.

Item 5. The Item of claim1, wherein, the instructions, when executed by the processor, further cause the processor to control the system to perform processing the captured first local image to emulate at least one of rotating, moving and zooming of the camera.

Item 6. The Item of claim1, where the instructions, when executed by the processor, further cause the processor to control the system to perform: after receiving the first remote FOV, receiving, from the remote system via the communication network, the second remote FOV of the remote subject that is different from the first remote FOV; in response to receiving the second remote FOV of the remote subject, carrying out one of: performing (i) causing, based on the received second remote FOV, the camera orienting unit to orient the camera in a second orientation, wherein the camera oriented in the second orientation has a second local FOV corresponding to the second remote FOV received from the remote system, and (ii) upon orienting the camera in the second orientation, causing the image capturing unit to capture a second local image through the display; performing (i) causing the image capturing unit to capture a third local image through the display, and (ii) processing, based on the received second remote FOV, the captured third local image to emulate at least one of rotating, moving and zooming of the camera; and performing (i) causing, based on the received second remote FOV, the camera orienting unit to orient the camera in a third orientation, (ii) upon orienting the camera in the third orientation, causing the image capturing unit to capture a fourth local image through the display, and (iii) processing, based on the received second remote FOV, the captured fourth local image to emulate at least one of rotating, moving and zooming of the camera; and transmitting, to the remote system via the communication network, the captured second local image, the processed third local image or the processed fourth local image.

Item 7. The Item of claim1, wherein, the instructions, when executed by the processor, further cause the processor to control the system to perform at least one of: detecting a position of the local subject with respect to the first local FOV; detecting a head-facing direction of the local subject with respect to the first local FOV; and detecting an eye-gazing direction of the local subject with respect to the first local FOV.

Item 8. The Item of claim7, wherein, for causing the camera orienting unit to orient the camera in the first orientation, the instructions, when executed by the processor, further cause the processor to control the system to perform: determining, based on the detected position of the local subject, that the local subject is not positioned at a center of the first local FOV; and adjusting the first orientation to offset a distance between the center of the first local FOV and the detected position of the local subject with respect to the first local FOV.

Item 9. The Item of claim7, wherein for causing the camera orienting unit to orient the camera in the first orientation, the instructions, when executed by the processor, further cause the processor to control the system to perform: determining, based on the detected head-facing direction or eye-gazing direction of the local subject, that the local subject is not gazing at a center of the first local FOV; and adjusting the first orientation to offset an angular difference between the center of the first local FOV and the detected head-facing direction or eye-gazing direction of the local subject.

Item 10. A non-transitory computer-readable medium storing executable instructions that, when executed by a processor, cause the processor to control a system to perform: receiving, from a remote system via a communication network, a first remote field of view (FOV) of a remote subject; orienting, based on the received first remote FOV, a camera in a first orientation, the camera positioned on a rear surface of a display, wherein the camera oriented in the first orientation has a first local FOV corresponding to the first remote FOV received from the remote system; upon orienting the camera in the first orientation, causing the camera to capture a local image through the display; and transmitting, to the remote system via the communication network, the captured first local image.

Item 11. A method of operating a system for capturing a local image for transmission to a remote system, the system comprising (i) a display having a front surface facing a local subject and a rear surface facing opposite to the front surface, and (ii) a camera positioned on the rear surface of the display and configured to capture a local image through the display, the method comprising: receiving, from a remote system via a communication network, a first remote field of view (FOV) of a remote subject; orienting, based on the received first remote FOV, the camera in a first orientation, wherein the camera oriented in the first orientation has a first local FOV corresponding to the first remote FOV received from the remote system; upon orienting the camera in the first orientation, causing the camera to capture a first local image through the display; and transmitting, to the remote system via the communication network, the captured first local image.

Item 12. The method of Item 11, wherein orienting the camera in the first orientation comprises rotating the camera or controlling zooming of the camera.

Item 13. The method of Item 12, wherein orienting the camera in the first orientation further comprises moving the camera on the rear surface of the display.

Item 14. The method of Item 13, wherein orienting the camera in the first orientation further comprises at least one of: moving the camera on a plane substantially parallel to the rear surface of the display; and moving the camera in a three-dimensional (3D) space on the rear surface of the display.

Item 15. The method of Item 13, further comprising processing the captured first local image to emulate at least one of rotating, moving and zooming of the camera.

Item 16. The method of Item 11, further comprising: after receiving the first remote FOV, receiving, from the remote system via the communication network, a second remote FOV of the remote subject that is different from the first remote FOV; in response to receiving the second remote FOV of the remote subject, carrying out one of: performing (i) orienting the camera in a second orientation, wherein the camera oriented in the second orientation has a second local FOV corresponding to the second remote FOV received from the remote system, and (ii) upon orienting the camera in the second orientation, causing the camera to capture a second local image through the display; performing (i) capturing a third local image through the display, and (ii) processing, based on the received second remote FOV, the captured third local image to emulate at least one of rotating, moving and zooming of the camera; and performing (i) orienting, based on the received second remote FOV, the camera in a third orientation, (ii) upon orienting the camera in the third orientation, causing the image capturing unit to capture a fourth local image through the display, and (iii) processing, based on the received second remote FOV, the captured fourth local image to emulate at least one of rotating, moving and zooming of the camera; and transmitting, to the remote system via the communication network, the captured second local image, the processed third local image or the processed fourth local image.

Item 17. The method of Item 11, further comprising: detecting a position of the local subject with respect to the first local FOV; detecting a head-facing direction of the local subject with respect to the first local FOV; and detecting an eye-gazing direction of the local subject with respect to the first local FOV.

Item 18. The method of Item 17, wherein orienting the camera in the first orientation comprises: determining, based on the detected position of the local subject, that the local subject is not positioned at a center of the first local FOV; and adjusting the first orientation to offset a distance between the center of the first local FOV and the detected position of the local subject with respect to the first local FOV.

Item 19. The method of Item 17, wherein orienting the camera in the first orientation further comprises: determining, based on the detected head-facing direction or eye-gazing direction of the local subject, that the local subject is not gazing at a center of the first local FOV; and adjusting the first orientation to offset adjusting the first orientation to offset an angular difference between the center of the first local FOV and the detected head-facing direction or eye-gazing direction of the local subject.

Item 20. A system for capturing a local image for transmission to a remote system, comprising: a processor; and a computer-readable medium in communication with the processor, the computer-readable medium comprising executable instructions that, when executed by the processor, cause the processor to control the system to perform functions of: receiving, from a remote system via a communication network, a remote field of view (FOV) of a remote subject; orienting, based on the received remote FOV, the camera in a first orientation, wherein the camera oriented in the first orientation has a local FOV corresponding to the remote FOV received from the remote system; detecting at least one of a position, head-facing direction and eye-gazing direction of the local subject with respect to the local FOV; determining, based on at least one of the detected position, head-facing direction and eye-gazing direction of the local subject with respect to the local FOV, that the local subject is not positioned at a center of the local FOV or the local subject is not gazing at the center of the local FOV; orienting the camera in a second orientation that offsets at least one of a distance between the center of the first local FOV and the detected position of the local subject and an angular difference between the center of the first local FOV and the detected head-facing direction or eye-gazing direction of the local subject; causing the camera to capture a local image with the second orientation; and transmitting, to the remote system via the communication network, the captured first local image.