Patent ID: 12217368

DETAILED DESCRIPTION

FIGS.1through8B, discussed below, and the various embodiments used to describe the principles of the present disclosure in this patent document are by way of illustration only and should not be construed in any way to limit the scope of the disclosure. Those skilled in the art will understand that the principles of the present disclosure may be implemented in any suitably-arranged system or device.

An electronic device, according to embodiments of the present disclosure, can include a personal computer (such as a laptop, a desktop), head mounted display, a workstation, a server, a television, an appliance, a client device, and the like. In certain embodiments, a client device can be a portable electronic device such as a portable communication device (such as a smartphone or mobile phone), a laptop, a tablet, an electronic book reader (such as an e-reader), a personal digital assistants (PDAs), a portable multimedia player (PMP), an MP3 player, a mobile medical device, a virtual reality headset, a portable game console, a camera, and a wearable device, among others. Additionally, the electronic device can be at least one of a part of a piece of furniture or building/structure, an electronic board, an electronic signature receiving device, a projector, or a measurement device. The electronic device is one or a combination of the above-listed devices. Additionally, the electronic device as disclosed herein is not limited to the above-listed devices and can include new electronic devices depending on the development of technology. It is noted that as used herein, the term “user” may denote a human or another device (such as an artificial intelligent electronic device) using the electronic device.

Virtual reality (VR) is a rendered version of a visual scene, where the entire scene is computer generated. For example, in a VR environment a user is fully immersed in a simulated digital environment. Augmented reality (AR) is an interactive experience of a real-world environment where objects that reside in the real-world environment are augmented with virtual objects, virtual information, or both. For example, in an AR environment, virtual objects can be overlaid on the real world. Mixed reality (MR) enables digital and real-world objects to co-exist and interact with one another in real-time. Extended reality (XR), as used herein refers to VR content, AR content, MR content, and the like. In certain embodiments, VR, AR and MR include both visual and audio experiences. A visual rendering is designed to mimic the visual stimuli, and if available audio sensory stimuli, of the real world as naturally as possible to an observer or user as the user moves within the limits defined by the application or the VR, AR, or MR scene. For example, VR places a user into immersive worlds that respond to detected head movements of a user. At the video level, VR is achieved by providing a video experience that covers as much of the field of view (FoV) as possible together with the synchronization of the viewing angle of the rendered video with the head movements.

Many different types of devices are able to provide the immersive experience associated with XR. One example device is a head-mounted display (HMD). An HMD represents one of many types of devices that provide XR experiences to a user. An HMD is a device that enables a user to view the XR scene and adjust the displayed content based on movements of the head of the user. Typically, an HMD relies either on a dedicated screen that is integrated into a device and connected with an external computer (tethered) or on a device, such as a smartphone, that is inserted into the HMD (untethered). The first approach utilizes one or more lightweight screens and benefits from a high computing capacity. In contrast, the smartphone-based systems utilize higher mobility and can be less expensive to produce. In both instances, the video experience generated is the same.

Volumetric video including XR content can be represented in different formats, such as panoramas or spheres. For example, volumetric video content can include a full 360° view, such that when a user changes their viewpoint they view different portions of the video. While the 360° video provides all around scenes, a user often views only a limited FoV. Various devices, such as an HMD can display portions of a spherical videos covering a full 360° view.

Embodiments of the present disclosure take into consideration that certain electronic devices, due to cost or mobility, lack the computational power required to render high-quality XR content on their own hardware. For example, certain electronic devices can struggle to provide the processing capabilities to handle receiving, decoding, and rendering an entire 360° video and then displaying only a portion of the 360° video.

Split rendering enables high-quality video to be rendered and displayed on low powered devices. That is, split rendering is the process where the decoding and rendering VR and volumetric video is split between two (or more) devices. For example, a split rendering system enables a server (such as an electronic device) to send a portion of the 360° video to a client device (or user device such as a head mounted display). By offloading certain aspects of video decoding and rendering to one or more servers in the local network or at the edge, and streaming the live-rendered content as video, a client device would merely need hardware decoders and network connections to enable high quality XR.

This is beneficial since the client device can lack the computational power required to render high-quality XR content but still provide high-quality XR content to a user. Additionally, since the client device displays only a portion of each 360° frame, it does not need to receive the entire 360° video, thereby reducing the bandwidth needed to transmit the video from its source to the client device. For example, a server can receive, from a client device, a viewpoint of that client device (corresponding to a FoV of a user) and provide to that client device the portion of the 360° video corresponding to that viewpoint. The client device then decodes and renders the received portion of the 360° video. Embodiments of the present disclosure take into consideration that if the user changes their view to view a different portion of the 360° video, the client device may not have the necessary video from the server to display.

In certain embodiments, split rendering for XR streaming uses two or more devices, that of an electronic device (such as a Multi-access Edge Computing (MEC) server) and a client device (such as a XR device, mobile phone, TV, AR glasses, head mounted display, and the like). Split rendering is the process where the decoding and rendering of a volumetric video is split between a client device and the electronic device. The electronic device (such as a MEC server) receives from client device the user viewpoint information. The viewpoint information of the user can be the viewing direction (yaw, pitch and roll) of the head as well as the eyes for 360° video. Alternatively, the viewpoint information of the user can be the head position (x, y, x) of the user coupled with the viewing direction for six degrees of freedom (DoF) volumetric content.

In certain embodiments, the electronic device (such as a MEC server) decodes and renders the volumetric video into a 2D frame corresponding to FoV visible on the client device (XR device) and compresses and transmits the 2D frame back to the client device (XR device). Since the user viewpoint may have changed between the time it was sent to MEC server and the time the MEC rendered 2D image is displayed on XR device, the MEC server can send a bigger image containing the extended FoV to allow for user viewpoint changes.

For example, a client device detects motion (including direction) of a user's head at time T1. The client device sends the motion information to the server. Alternatively, the client device can send multiple viewpoints over successive times instances, enabling the server to determine the speed and direction of the motion of the client device. The server determines a FoV (based on the received motion information or multiple viewpoints). The server also determines an extended FoV (which is larger than the FoV) based on the motion of the user's head at time T1. The extended FoV covers a larger area than the actual FoV. The server, then sends the extended FoV to the client device. After the client device receives the extended FoV, the client device detects users motion a time T2. The client device then determines a FoV based on the detected motion at time T2. Since the extended FoV is larger than the FoV the client device identifies a portion of the extended FoV to display. Then at time T3, the client device displays the determined FoV. It is noted that the time difference between T2 and T3 is related to motion to photon (MTP) latency.

Embodiments of the present disclosure take into consideration that when determining an extended FoV, the server can use a predetermined constant to control how much the FoV should be extended. When determining the size of the extended FoV the server balances two conflicting factors. First, as the size extended FoV increases, the client device is forced to perform more processing, reducing the effectiveness of split rendering. Second, if the size of the extended FoV is not large enough, any change in the viewpoint of the user could cause the client device to not have the necessary video content to render and display. For example, when the user suddenly moves their head, such as to view an entirely new portion of the 360° video, the client device may not have content associated with that portion of the 360° video. By increasing the size of the extended FoV to avoid missing data due to changes in viewpoint, can lead to increased bitrate since the frame size increases.

Accordingly, embodiments of the present disclosure describe extending a FoV based on head speed (velocity). Embodiments of the present disclosure also describe asymmetric FoV extensions based on head speed (velocity) and direction. For example, the FoV can be extended asymmetrically along the direction of the head motion to reduce the probability of the user viewport falling outside the extended FoV. Embodiments of the present disclosure further describe compacting (scaling) the extended FoV based on head speed (velocity). For example, the extended FoV sub-image can be compacted by a scale factor based on the head motion. If the head velocity is very high a stronger scaling (lower scaling factor) can be used. If the head velocity is low, a weak scaling (higher scaling factor) or no scaling can be used. The final extended FoV image is then compressed and sent to the client device. metadata needed to invert the scaling is also sent from the electronic device to the client device. The client device can then decompress the received image use the metadata to invert the scaling process.

Additionally, embodiments of the present disclosure describe event based lookahead for split rendering, such that the server generates video content associated with a future event in addition to an extended FoV corresponding to a current FoV. For example, a sub-image can also be selected based on events which might draw the attention of the user, such as introduction of an enemy in the scene during the computer game.

FIG.1illustrates an example communication system100in accordance with an embodiment of this disclosure. The embodiment of the communication system100shown inFIG.1is for illustration only. Other embodiments of the communication system100can be used without departing from the scope of this disclosure.

The communication system100includes a network102that facilitates communication between various components in the communication system100. For example, the network102can communicate internet protocol (IP) packets, frame relay frames, Asynchronous Transfer Mode (ATM) cells, or other information between network addresses. The network102includes one or more local area networks (LANs), metropolitan area networks (MANs), wide area networks (WANs), all or a portion of a global network such as the Internet, or any other communication system or systems at one or more locations.

In this example, the network102facilitates communications between a server104and various client devices106-116. The client devices106-116may be, for example, a smartphone, a tablet computer, a laptop, a personal computer, a wearable device, a head mounted display (HMD), or the like. The server104can represent one or more servers. Each server104includes any suitable computing or processing device that can provide computing services for one or more client devices, such as the client devices106-116. Each server104could, for example, include one or more processing devices, one or more memories storing instructions and data, and one or more network interfaces facilitating communication over the network102. As described in more detail below, the server104can generate the immersive content that is captured by one or more of the client devices106-116.

Each client device106-116represents any suitable computing or processing device that interacts with at least one server (such as the server104) or other computing device(s) over the network102. The client devices106-116include a desktop computer106, a mobile telephone or mobile device108(such as a smartphone), a PDA110, a laptop computer112, a tablet computer114, and an HMD116. However, any other or additional client devices could be used in the communication system100. Smartphones represent a class of mobile devices108that are handheld devices with mobile operating systems and integrated mobile broadband cellular network connections for voice, short message service (SMS), and Internet data communications. The HMD116can display a 360° scene including one or more animated scans. The HMD116can display a 360° scene of immersive content that is captured by multiple devices.

In this example, some client devices108-116communicate indirectly with the network102. For example, the mobile device108and PDA110communicate via one or more base stations118, such as cellular base stations or eNodeBs (eNBs). Also, the laptop computer112, the tablet computer114, and the HMD116communicate via one or more wireless access points120, such as IEEE 802.11 wireless access points. Note that these are for illustration only and that each client device106-116could communicate directly with the network102or indirectly with the network102via any suitable intermediate device(s) or network(s). In certain embodiments, the server104or any client device106-116can be used to generate the immersive content and transmit the content to another client device such as any client device106-116.

In certain embodiments, any of the client devices106-114transmit information securely and efficiently to another device, such as, for example, the server104. Also, any of the client devices106-116can trigger the information transmission between itself and the server104. Any of the client devices106-114can function as a VR display when attached to a headset via brackets, and function similar to HMD116. For example, the mobile device108when attached to a bracket system and worn over the eyes of a user can function similarly as the HMD116. The mobile device108(or any other client device106-116) can trigger the information transmission between itself and the server104.

In certain embodiments, any of the client devices106-116or the server104can generate immersive content, transmit the immersive content, receive and render generate the immersive content, or a combination thereof. For example, the mobile device108can capture video of an event and transfer the video to another client device or to a server (such as the server104) to generate immersive content. Additionally, any of the client devices106-116and the server can work together to perform split rendering.

AlthoughFIG.1illustrates one example of a communication system100, various changes can be made toFIG.1. For example, the communication system100could include any number of each component in any suitable arrangement. In general, computing and communication systems come in a wide variety of configurations, andFIG.1does not limit the scope of this disclosure to any particular configuration. WhileFIG.1illustrates one operational environment in which various features disclosed in this patent document can be used, these features could be used in any other suitable system.

FIGS.2and3illustrate example electronic devices in accordance with an embodiment of this disclosure. In particular,FIG.2illustrates an example server200, and the server200could represent the server104inFIG.1. The server200can represent one or more, local servers, remote servers, clustered computers, and components that act as a single pool of seamless resources, a cloud-based server, and the like. The server200can be accessed by one or more of the client devices106-116ofFIG.1or another server.

The server200can represent one or more local servers, one or more compression servers, one or more media processing services, one or more encoding servers, an MEC server, and the like. As shown inFIG.2, the server200includes a bus system205that supports communication between at least one processing device (such as a processor210), at least one storage device215, at least one communication interface220, and at least one input/output (I/O) unit225.

The processor210executes instructions that can be stored in a memory230. The processor210can include any suitable number(s) and type(s) of processors or other devices in any suitable arrangement. Example types of processors210include microprocessors, microcontrollers, digital signal processors, field programmable gate arrays, application specific integrated circuits, and discrete circuitry. In certain embodiments, the processor210can combine various streams of media of the same event to create an immersive content.

The memory230and a persistent storage235are examples of storage devices215that represent any structure(s) capable of storing and facilitating retrieval of information (such as data, program code, or other suitable information on a temporary or permanent basis). The memory230can represent a random-access memory or any other suitable volatile or non-volatile storage device(s). For example, the instructions stored in the memory230can include instructions for creating immersive content from multiple data streams. The instructions stored in the memory230can also include instructions for rendering a 360° scene, as viewed through a VR headset, such as HMD116ofFIG.1. The persistent storage235can contain one or more components or devices supporting longer-term storage of data, such as a read only memory, hard drive, Flash memory, or optical disc.

The communication interface220supports communications with other systems or devices. For example, the communication interface220could include a network interface card or a wireless transceiver facilitating communications over the network102ofFIG.1. The communication interface220can support communications through any suitable physical or wireless communication link(s). For example, the communication interface220can transmit immersive content to another device such as one of the client devices106-116.

The I/O unit225allows for input and output of data. For example, the I/O unit225can provide a connection for user input through a keyboard, mouse, keypad, touchscreen, or other suitable input device. The I/O unit225can also send output to a display, printer, or other suitable output device. Note, however, that the I/O unit225can be omitted, such as when I/O interactions with the server200occur via a network connection.

Note that whileFIG.2is described as representing the server104ofFIG.1, the same or similar structure could be used in one or more of the various client devices106-116. For example, a desktop computer106or a laptop computer112could have the same or similar structure as that shown inFIG.2.

FIG.3illustrates an example electronic device300, and the electronic device300could represent one or more of the client devices106-116inFIG.1. The electronic device300can be a mobile communication device, such as, for example, a mobile station, a subscriber station, a wireless terminal, a desktop computer (similar to the desktop computer106ofFIG.1), a portable electronic device (similar to the mobile device108, the PDA110, the laptop computer112, the tablet computer114, or the HMD116ofFIG.1), and the like. In certain embodiments, one or more of the client devices106-116ofFIG.1can include the same or similar configuration as the electronic device300. In certain embodiments, the electronic device300is captures content of an event. For example, the electronic device300is usable with data transfer, capturing content based on various parameters, and media rendering applications.

As shown inFIG.3, the electronic device300includes an antenna305, a radio-frequency (RF) transceiver310, transmit (TX) processing circuitry315, a microphone320, and receive (RX) processing circuitry325. The RF transceiver310can include, for example, a RF transceiver, a BLUETOOTH transceiver, a WI-FI transceiver, a ZIGBEE transceiver, an infrared transceiver, and various other wireless communication signals. The electronic device300also includes a speaker330, a processor340, an input/output (I/O) interface (IF)345, an input350, a display355, a memory360, and a sensor(s)365. The memory360includes an operating system (OS)361and one or more applications362.

The RF transceiver310receives, from the antenna305, an incoming RF signal transmitted from an access point (such as a base station, WI-FI router, or BLUETOOTH device) or other device of the network102(such as a WI-FI, BLUETOOTH, cellular, 5G, LTE, LTE-A, WiMAX, or any other type of wireless network). The RF transceiver310down-converts the incoming RF signal to generate an intermediate frequency or baseband signal. The intermediate frequency or baseband signal is sent to the RX processing circuitry325that generates a processed baseband signal by filtering, decoding, and/or digitizing the baseband or intermediate frequency signal. The RX processing circuitry325transmits the processed baseband signal to the speaker330(such as for voice data) or to the processor340for further processing (such as for web browsing data).

The TX processing circuitry315receives analog or digital voice data from the microphone320or other outgoing baseband data from the processor340. The outgoing baseband data can include web data, e-mail, or interactive video game data. The TX processing circuitry315encodes, multiplexes, and/or digitizes the outgoing baseband data to generate a processed baseband or intermediate frequency signal. The RF transceiver310receives the outgoing processed baseband or intermediate frequency signal from the TX processing circuitry315and up-converts the baseband or intermediate frequency signal to an RF signal that is transmitted via the antenna305.

The processor340can include one or more processors or other processing devices. The processor340can execute instructions that are stored in the memory360, such as the OS361in order to control the overall operation of the electronic device300. For example, the processor340could control the reception of forward channel signals and the transmission of reverse channel signals by the RF transceiver310, the RX processing circuitry325, and the TX processing circuitry315in accordance with well-known principles. The processor340can include any suitable number(s) and type(s) of processors or other devices in any suitable arrangement. For example, in certain embodiments, the processor340includes at least one microprocessor or microcontroller. Example types of processor340include microprocessors, microcontrollers, digital signal processors, field programmable gate arrays, application specific integrated circuits, and discrete circuitry.

The processor340is also capable of executing other processes and programs resident in the memory360, such as operations that receive and store data. The processor340can move data into or out of the memory360as required by an executing process. In certain embodiments, the processor340is configured to execute the one or more applications362based on the OS361or in response to signals received from external source(s) or an operator. Example, applications362can include a VR or AR application, a camera application (for still images and videos), a video phone call application, an email client, a social media client, a SMS messaging client, a virtual assistant, and the like.

The processor340is also coupled to the I/O interface345that provides the electronic device300with the ability to connect to other devices, such as client devices106-114. The I/O interface345is the communication path between these accessories and the processor340.

The processor340is also coupled to the input350and the display355. The operator of the electronic device300can use the input350to enter data or inputs into the electronic device300. The input350can be a keyboard, touchscreen, mouse, track ball, voice input, or other device capable of acting as a user interface to allow a user to interact with the electronic device300. For example, the input350can include voice recognition processing, thereby allowing a user to input a voice command. In another example, the input350can include a touch panel, a (digital) pen sensor, a key, or an ultrasonic input device. The touch panel can recognize, for example, a touch input in at least one scheme, such as a capacitive scheme, a pressure sensitive scheme, an infrared scheme, or an ultrasonic scheme. The input350can be associated with the sensor(s)365and/or a camera by providing additional input to the processor340. In certain embodiments, the sensor365includes one or more inertial measurement units (IMUs) (such as accelerometers, gyroscope, and magnetometer), motion sensors, optical sensors, cameras, pressure sensors, heart rate sensors, altimeter, and the like. The input350can also include a control circuit. In the capacitive scheme, the input350can recognize touch or proximity.

The display355can be a liquid crystal display (LCD), light-emitting diode (LED) display, organic LED (OLED), active matrix OLED (AMOLED), or other display capable of rendering text and/or graphics, such as from websites, videos, games, images, and the like. The display355can be sized to fit within an HMD. The display355can be a singular display screen or multiple display screens capable of creating a stereoscopic display. In certain embodiments, the display355is a heads-up display (HUD). The display355can display 3D objects and immersive content.

The memory360is coupled to the processor340. Part of the memory360could include a RAM, and another part of the memory360could include a Flash memory or other ROM. The memory360can include persistent storage (not shown) that represents any structure(s) capable of storing and facilitating retrieval of information (such as data, program code, and/or other suitable information). The memory360can contain one or more components or devices supporting longer-term storage of data, such as a read only memory, hard drive, Flash memory, or optical disc. The memory360also can contain media content. The media content can include various types of media such as images, videos, three-dimensional content, VR content, AR content, immersive content, and the like.

The electronic device300further includes one or more sensors365that can meter a physical quantity or detect an activation state of the electronic device300and convert metered or detected information into an electrical signal. For example, the sensor365can include one or more buttons for touch input, a camera, a gesture sensor, an IMU sensors (such as a gyroscope or gyro sensor and an accelerometer), an eye tracking sensor, an air pressure sensor, a magnetic sensor or magnetometer, a grip sensor, a proximity sensor, a color sensor, a bio-physical sensor, a temperature/humidity sensor, an illumination sensor, an Ultraviolet (UV) sensor, an Electromyography (EMG) sensor, an Electroencephalogram (EEG) sensor, an Electrocardiogram (ECG) sensor, an IR sensor, an ultrasound sensor, an iris sensor, a fingerprint sensor, a color sensor (such as a Red Green Blue (RGB) sensor/camera), a depth sensor, a D-RGB sensor (depth Red Green Blue sensor/camera), and the like. The sensor365can further include control circuits for controlling any of the sensors included therein.

As discussed in greater detail below, one or more of these sensor(s)365may be used to control a user interface (UI), detect UI inputs, determine the orientation and facing the direction of the user for three-dimensional content display identification, and the like. Any of these sensor(s)365may be located within the electronic device300, within a secondary device operably connected to the electronic device300, within a headset configured to hold the electronic device300, or in a singular device where the electronic device300includes a headset.

AlthoughFIGS.2and3illustrate examples of electronic devices, various changes can be made toFIGS.2and3. For example, various components inFIGS.2and3could be combined, further subdivided, or omitted and additional components could be added according to particular needs. As a particular example, the processor340could be divided into multiple processors, such as one or more central processing units (CPUs) and one or more graphics processing units (GPUs). In addition, as with computing and communication, electronic devices and servers can come in a wide variety of configurations, andFIGS.2and3do not limit this disclosure to any particular electronic device or server.

FIG.4Aillustrates a block diagram of an example environment-architecture400in accordance with an embodiment of this disclosure.FIGS.4B and4Cillustrate example video types in accordance with an embodiment of this disclosure. The embodiment ofFIGS.4A,4B, and4Care for illustration only. Other embodiments can be used without departing from the scope of this disclosure.

The example environment-architecture400, as shown inFIG.4A, includes an electronic device410and a client device420in communication over a network402. In certain embodiments, the electronic device410is a server (such as a MEC server) and the client device420is an XR device such as an HMD.

The network402can be the same as or similar to the network102ofFIG.1. In certain embodiments, the network402represents a “cloud” of computers interconnected by one or more networks, where the network is a computing system utilizing clustered computers and components that act as a single pool of seamless resources when accessed. Also, in certain embodiments, the network402is connected with one or more servers (such as the server104ofFIG.1, the server200), one or more electronic devices (such as the client devices106-116ofFIG.1, the electronic device300), the electronic device410, and a client device420. Further, in certain embodiments, the network402can be connected to an information repository (not shown) that contains XR media content that can be encoded by the electronic device410, rendered and displayed to a user by the client device420.

In certain embodiments, the electronic device410and the client device420can represent the server104, one of the client devices106-116ofFIG.1, the server200ofFIG.2, the electronic device300ofFIG.3, or another suitable device. In certain embodiments, the electronic device410can be a “cloud” of computers interconnected by one or more networks, where each is a computing system utilizing clustered computers and components to act as a single pool of seamless resources when accessed through the network402. In some embodiments, a portion of the components included in the electronic device410or the client device420can be included in different devices, such as multiple servers104or200, multiple client devices106-116, or other combination of different devices. In certain embodiments, the electronic device410and the client device420work together when performing split rendering.

The electronic device410, receives 3D media content, such as a 360° video, from another device such as a server (similar to the server104ofFIG.1, the server200ofFIG.2), an information repository (such as a database), or one of the client devices106-116. In certain embodiments, the electronic device410can receive media content from multiple cameras and stitch the content together to generate a 3D scene that includes one or more point clouds. Additionally, the electronic device410can receive user data from the client device420via the network402. The user data can include a viewpoint of the user and motion data.

The client device420can include a motion sensor (similar to the sensor365ofFIG.3) and a display (similar to the display355ofFIG.3). The motion sensor can be used to detect the viewpoint information of the user. For example, the motion sensor can detect the viewing direction, including yaw, pitch and roll of the head. For another example, the motion sensor can be used to identify a position of the head coupled with a viewing direction. The display of the client device420can be similar to the display355ofFIG.3.

FIGS.4B and4Cillustrate example video types, in accordance with an embodiment of this disclosure. Video content can be immersive 360° video. As shown inFIG.4B, the video content can be a video equirectangular projection video430As shown inFIG.4C, the video content can be cube map video440.

Since the user using a client device420can view a portion of the video at a time, the electronic device410can transmit the portion of the video corresponding to the viewpoint of the user. Since the viewpoint of the user can change from a time when the viewpoint information is transmitted to the electronic device410to when the client device receives the portion of the video content, the electronic device410can transmit an extended FoV that includes additional video content to accommodate slight changes in the viewpoint.

Embodiments of the present disclosure take into consideration that extended FoV can be increased to avoid missed data due to fast head motion. However as the size extended FoV increases, can also leads to increased bitrate (since video frame size increases). Therefore, embodiments of the present disclosure describe that the extent that the FoV is extended is based on head velocity. For example, the FoV is extended along the direction of the head motion as shown inFIGS.4B and4C. This extended FoV image is then transmitted by the electronic device410(such as a server or a MEC) to the client device420(such as an XR device). By extending the FoV in certain direction reduces both (i) the probability of the viewport falling outside the extended FoV and avoiding missing data and (ii) the bitrate since the video frame size can decrease.

FIG.4Billustrate example video, equirectangular projection video430. The box corresponding to the viewpoint432represents a FoV and the box corresponding to the extended FoV434is based on the motion436of the user. Similarly,FIG.4Cillustrates an example cube map video440. The box corresponding to the viewpoint442represents a FoV and the box corresponding to the extended FoV444is based on the motion446of the head of the user. That is, the extension of the FoV along the direction of the head motion (such as motion436and446) can lead to asymmetric FoV extensions (such as the extended FoV434and444), based on the type of video.

AlthoughFIGS.4A,4B, and4Cillustrate the environment-architecture400and various video types of various changes can be made toFIGS.4A,4B, and4C. For example, any number of electronic devices or client devices can be included environment-architecture400. For another example, other types of video formats can be used for split rendering.

FIG.5Aillustrates an example method500for split rendering in accordance with an embodiment of this disclosure.FIGS.5B,5C, and5Dillustrate diagrams520,530, and540respectively, for extending the FoV based on motion in accordance with an embodiment of this disclosure.

The method500is described as implemented by any one of the client device106-116ofFIG.1, the server104ofFIG.1, or the electronic device410, and can include internal components similar to that of server200ofFIG.2and the electronic device300ofFIG.3. However, the method500as shown inFIG.5Acould be used with any other suitable electronic device and in any suitable system. The embodiments ofFIGS.5A,5B,5C and5Dare for illustration only. Other embodiments can be used without departing from the scope of this disclosure.

In step502, the electronic device410(such as a MEC server) obtains user data from a client device420(such as an XR device). The user data can include the viewpoint of the client device420. The user data can include motion data such as a speed (velocity) and direction of motion as detected by the client device420. In certain embodiments, the user data includes multiple viewpoints in addition to or in alternative of the motion data.

In step504, the electronic device410identifies the FoV centered around the user viewpoint. In step506, the electronic device410identifies the head velocity. In certain embodiment, the electronic device410uses historical data from the previous user viewpoints, to determine the velocity of the client device420. The head velocity could be based on simple regression or a complicated neural network. In other embodiments, the electronic device410receives motion data from a sensor of the client device420to determine the velocity of the client device420.

In step508, the electronic device410identifies the extended FoV based on the current FoV (from step504) and the identified velocity (step506). In step510, the electronic device410renders the volumetric video that lies in the extended FoV. In step512, the electronic device410compresses the resulting rendered video frame using a video encoder. In step514, the electronic device transmits the compressed video to the client device420.

FIG.5Billustrates the diagram520for calculating the corners of the extended FoV. As illustrated, FoV_L, FoV_R, FoV_T, and FoV_B are the respective left, right, top bottom corners of the FoV, and eFoV_L, eFoV_R, eFoV_T, and eFoV_B are the respective left, right, top bottom corners of the extended FoV. Equation (1) describes the predefined head velocity in terms of FoV X-axis and Y-axis directions or yaw and pitch, respectively.
hv=(hvx,hvy)  (1)

Equations (2)-(5), below, describe the corners of the extended FoV.
eFoV_L=FoV_L−b*hvx(2)
eFoV_R=FoV_R+c*hvx(3)
eFoV_T=FoV_T−d*hvy(4)
eFoV_B=FoV_B+e*hvy(5)
Here, the variables b, c, d, and e are predetermined constants that control how much of the FoV should be extended based on head velocity. That is, the head velocity in the X or Y directions control the size of the extended FoV.

The diagram520ofFIG.5Bshows a 360 degree image522and a FoV526. The FoV526is a portion of the entire 360 degree image522. The extended FoV524is based on the Equations (2)-(5). For six DOF scenes, the volumetric scene is rendered with FoV526set to the extended FoV524.

The diagram530ofFIG.5Cillustrates the electronic device410determining the extended FoV534based on the viewpoint532and the motion536that are obtained from the client device420. The diagram540ofFIG.5Dillustrates the client device420rendering a portion542of the extended FoV534that is received from the electronic device410. For example, at a given time, T1, the electronic device410receives user data including the viewpoint432. The electronic device can also receive the motion data (motion436) from the client device420, multiple viewpoints for determining the motion of the client device, or both. The electronic device410determines the size of the extended FoV based on the received user data, using the Equations (2)-(5). At time T2, the electronic device410sends the extended FoV to the client device420. Upon the client device420receiving the extended FoV534from the electronic device410, the client device420determines a portion542of the extended FoV534that corresponds to the current FoV at time T3. It is noted that time T2and time T3occurs after the time T1. As illustrated, the portion542ofFIG.5Dis both higher and to the left as compared to the viewpoint532ofFIG.5C, which roughly corresponds to the direction of the motion536.

AlthoughFIG.5Aillustrates one example of the method500, various changes may be made toFIG.5A. For example, while shown as a series of steps, various steps inFIG.5Acould overlap, occur in parallel, or occur any number of times. The embodiments ofFIGS.5A-5Dare for illustration only. Other embodiments can be used without departing from the scope of this disclosure.

FIG.6Aillustrates an example method600for scaling a split rendering in accordance with an embodiment of this disclosure.FIGS.6B and6Cillustrates diagrams620and630, respectively, for scaling an extended FoV based on motion in accordance with an embodiment of this disclosure.FIG.6Dillustrates an example method640for decoding a split rendering in accordance with an embodiment of this disclosure.

The methods600and640are described as implemented by any one of the client device106-116ofFIG.1, the server104ofFIG.1, the electronic device410ofFIG.4A, or the client device420ofFIG.4Aand can include internal components similar to that of server200ofFIG.2and the electronic device300ofFIG.3. However, the method600as shown inFIG.6Aand the method640as shown inFIG.6Dcould be used with any other suitable electronic device and in any suitable system. The embodiments ofFIGS.6A-6Dare for illustration only. Other embodiments can be used without departing from the scope of this disclosure.

For ease of explanation, the method600ofFIG.6Ais described as being performed by the electronic device410ofFIG.4Aand the method640ofFIG.6Dis described as being performed by the client device420ofFIG.4A.

In addition to extending the FoV based on the speed and direction of the motion data, embodiments of the present disclosure describe that the extended video can be compressed and scaled differently than the video corresponding to the FoV that is identified in step504ofFIG.5A. For example, the extended FoV (such as the extended FoV444ofFIG.4C) can be compacted so as to not increase the size of the video frame. Scaling the video corresponding to the FoV or the extended FoV can reduce the special resolution.

The method600as shown inFIG.6Ais similar to the method500ofFIG.5A. Therefore, for brevity the steps that are repeated in bothFIGS.5A and6Aare omitted here in the description ofFIG.6A.

In step602, the electronic device410generates a scaling factor for the video corresponding to the extended FoV. After the video is rendered in step510, the electronic device410uses the identified head velocity (of step506) to generate the scaling factor. The amount of scaling is based on the head speed (velocity) as determined by the motion data or using multiple viewpoints. For example, the volumetric video is rendered, the extended FoV sub-image is scaled appropriately along the direction of the head motion. Different scaling factors can be used such as ½, ⅓, ⅔, ¾, and the like. Other scale factors can be used as well. The scale factors can be adapted based on the head velocity so as to maintain the final size of the extended FoV region. For example, if the head velocity is very high a stronger scaling (i.e. lower scaling factor) can be used. If the head velocity is low, a weak scaling (i.e. higher scaling factor) or no scaling can be used.

Syntax (1) below describes an example process for generating the scaling factor, S, for the extended FoV.

Syntax(1)if (| hv | < t_slow)s = 1; /* no scaling */else if (| hv | < t_normal)s = p; /* low amount of scaling */elses = q; /* high amount of scaling */

Here, q and p are predetermined constants, where q≤p≤1. Additionally, t slow and t normal are thresholds used to determine whether the head speed is considered slow or fast. Moreover, hv is the magnitude of the head velocity. It can be based on max(abs(hvx), abs(hvy)), sqrt(hvx*hvx, hvy*hvy), and the like.

In certain embodiments, the scaling factor can be different for horizontal and vertical directions.

In certain embodiments, the video corresponding to the extended FoV (such as the extended FoV444ofFIG.4C) can be compacted such as illustrated inFIG.6C. In certain embodiments, the video corresponding to the original FoV sub-image (such as the viewpoint442ofFIG.4C) is not compacted. In other embodiments, the video corresponding to the original FoV sub-image (such as the viewpoint442ofFIG.4C) is compacted with scale factors that are different than those used for the extended FoV (such as the extended FoV444ofFIG.4C).

Compacting the video corresponding to the extended FoV (and not the identified FoV) can lower the probability of missing data since the electronic device410can include more video content. Even though the video corresponding to the extended FoV will have a lower spatial resolution than the identified FoV, this may not reduce the visual quality of the image since the extended FoV image would likely remain in the peripheral vision of the user.

In step512a, the electronic device410compresses the resulting rendered video frame using a video encoder. For example, compacted extended FoV and the FoV image are compressed using a video encoder. In step514a, the electronic device transmits the compressed video back to the client device420. In step605, the electronic device410transmits metadata to the client device420. The metadata can include scaling factor, constants (b, c, d, e) and head velocity (hvx,hvy), and the like. In certain embodiments, the metadata can also include the head velocity. The scaling factor that is transmitted to the client device420, enables the client device420to invert the scaling.

In certain embodiments, the scaling factor and the portions of the video that are transmitted to the client device420can be multiplexed together.

FIG.6Billustrates the diagram620for scaling the extended FoV624. As illustrated in the diagram620, the extended FoV is scaled by a factor S, while the FoV426is not scaled. The input image (as shown in the diagram520FIG.4B) has a width based on the difference between eFoV_R and eFoV_L, and a height that is based on the difference between eFoV_B and eFoV_T. The scaled image as shown in the diagram620has a width that is sFoV_R and a height that is sFoV_B.

FIG.6Cillustrates the diagram630for scaling the extended FoV. The video content of the diagram630is similar to the video content ofFIG.4C. The diagram630ofFIG.6Cillustrates the electronic device410determining the extended FoV634abased on the viewpoint632and the motion636that are received from the client device420. Based on the motion636, the electronic device410determines the scaling factor and generates a video corresponding to the compacted extended FoV634b, which can be transmitted to the client device420, along with the scaling factor used when generating the compacted extended FoV634b. The amount that the video corresponding to the extended FoV634ais compacted is based on the motion of the client device420. In certain embodiments, the electronic device410can also compact the video corresponding to the viewpoint632using the same or different scaling factor that is used for the video corresponding to the extended FoV634a.

The method640as shown inFIG.6Ddescribes the client device420decoding, descaling, and rendering a portion of the video content on a display for a user. The client device420obtains information650from the electronic device410. The obtained information can include compressed and scaled bitstream652and metadata654. The compressed and scaled bitstream652represents a portion of the video content. The metadata654includes the scaling factor, the extended FoV constants. The metadata654can also include the speed (velocity) of the client device420. In certain embodiments, the client device420previously sent the speed (velocity) to the electronic device410. In certain embodiments, if the client device420sent viewpoint information to the electronic device410, then the electronic device410determined the speed (velocity) of the client device420.

In step642, the client device420decompresses a received compressed and scaled bitstream652. In step644, the client device420uses the metadata654to do inverse scaling. The client device420using the most recent head motion information656, selects a portion of the FoV and renders it on a display (step648).

AlthoughFIGS.6A and6Dillustrate the methods600and640, various changes may be made toFIGS.6A and6D. For example, while shown as a series of steps, various steps inFIGS.6A and6Dcould overlap, occur in parallel, or occur any number of times. The embodiments ofFIGS.6A-6Dare for illustration only. Other embodiments can be used without departing from the scope of this disclosure.

FIG.7Aillustrates an example method700for event based lookahead split rendering in accordance with an embodiment of this disclosure.FIG.7Billustrates an example method720for decoding an event based lookahead split rendering in accordance with an embodiment of this disclosure.FIG.7Cillustrates a diagram740of a lookahead FoV corresponding to a future event in accordance with an embodiment of this disclosure.

In certain embodiments, another extended FoV can be transmitted to from the electronic device410to the client device420based on events that occur within of the video content. Contents with predefined stories or logics have predefined events, which might draw the attention of the user, such as when an enemy is introduced into a scene of a movie or game. For example, users who watch any episodic contents or play computer games there will be some events the viewpoint of a user may be directed to such as introduction of new objects, modification of some existing objects, or removal of any existing objects and the like.

Event based look enables the electronic device410to identify the events that will happen in the future within the MTP latency but in an area of the video content that is beyond the extended FoV. In certain embodiments, the events could require the user to view them. In other embodiments, if such event matches with a user's interest (as specified in user interest data), the electronic device410can render the predicted events before it happens and include the region showing such events as a part of the extended FoV even if the area for such events is not direct extension of extended FoV calculated based on MTP latency. That is, the additional region can be separate from the extended FoV. The electronic device410can access the information about the contents and find the events that will occur in the future based on such content information. The electronic device410can then decide whether to include some, all, or none of the events as an additional extended FoV according to the preferences the users or the general behavior of the users.

In certain embodiments, if the electronic device410knows of an event that will occur in the video content in the near future (beyond current MTP latency), then the electronic device410can render the predicted event before it happens and sent it to the client device420for later viewing. Upon receiving the additional extended FoV associated with the future event, the client device420can store the data for later use. For example, the electronic device410can render the predicted events which will happen in the near future before it happens and send it to the client device420for later use while it is sending the video for current FoV and extended FoV based on MPT latency. The client device420will receive such additional video and store it in the local storage and if the user moves to the area the events happens when the events actually happens then the encodes of the electronic device410transmits the difference between the previously sent lookahead FoV video and scene of the actual events happening and send it to the client device420as well, then the client decode and display combined video.

The methods700and720are described as implemented by any one of the client device106-116ofFIG.1, the server104ofFIG.1, the electronic device410ofFIG.4A, or the client device420ofFIG.4Aand can include internal components similar to that of server200ofFIG.2and the electronic device300ofFIG.3. However, the method700as shown inFIG.7Aand the method720as shown inFIG.7Bcould be used with any other suitable electronic device and in any suitable system. The embodiments ofFIGS.7A-7Care for illustration only. Other embodiments can be used without departing from the scope of this disclosure.

For ease of explanation, the method700ofFIG.7Ais described as being performed by the electronic device410ofFIG.4Aand the method720ofFIG.7Bis described as being performed by the client device420ofFIG.4A.

In step702ofFIG.7A, the electronic device410obtains user data. the user data can include a viewpoint, multiple viewpoints, motion data, or a combination thereof. The user data can also include preferences of the user, such as events in the video content that the user prefers to view. In step704, the electronic device410identifies a region of the video centered around the user viewpoint. In step706, the electronic device410generates a video corresponding to the FoV based on the identified region (of step704). In step708athe electronic device410compresses the resulting rendered video frame using a video encoder.

In step710, the electronic device410identifies a future event that will occur in in the video content. For example, the electronic device410accesses information about the content. The electronic device410can predict new events based on the information about the content and user data (such as the events of interest to the user). The electronic device410then decides the future events to be included in the additional video based on the user data.

In step712, the electronic device410renders lookahead FoV to cover the movement of the user to the future event if the lookahead FoV is not connected to the current FoV or in the extended FoV. This can be based on MTP latency where the amount of lookahead time is based on the user data. In step708b, the electronic device410encodes the lookahead FoV as a separate video from the current FoV (which was encoded in step708a).

In step714, the electronic device410can multiplex the encoded FoV (of step708a) and the encoded lookahead FoV (of step708b). In certain embodiments, the electronic device410can statistically multiplex the encoded FoV (of step708a) and the encoded lookahead FoV (of step708b) to keep the bandwidth consistent. In step716, the electronic device410can transmit the video to the client device.

In certain embodiments, when the lookahead FoV is to be rendered, the electronic device410encodes difference between the video sent as a lookahead FoV and the final FoV according to the current user data. The difference between the lookahead FoV and the final FoV is referred to as a delta FoV. Thereafter, when the electronic device410triggers playback of a lookahead FoV, then the client device420renders and displays the previously received lookahead FoV and merge it with delta FoV.

In step722aofFIG.7B, the client device420obtains from the electronic device410, the delta FoV. In step722b, the client device420also obtains a trigger for the lookahead FoV. The trigger can be the lookahead FoV with a time stamp. In step724aand724b, the client device420decodes the delta FoV and the lookahead FoV, respectively. In step726, the client device420merges the two videos together. In step728, the client device420renders and displays the merged video for the user to view.

The diagram740ofFIG.7Cincludes a 360° video. The electronic device410parses the video to find any events that occur within the video based on content information. The diagram740includes viewpoint742and an extended FoV744. The viewpoint742can represent a current FoV and the extended FoV744can be based on the head motion of a user. The electronic device410can identify, from the content information, that an event750(here, the event is a person cycling on the basketball court) will occur at a time T. The electronic device410can then generate a lookahead FoV752and transmit it to the client device420. In certain embodiments, the electronic device410can determine whether to send the lookahead FoV752based on preferences of the user that are received from the client device. The electronic device410can separately encode the lookahead FoV752and the extended FoV744when sending the content to the client device420.

In certain embodiments, the electronic device410transmits both the extended FoV744and the lookahead FoV752. In certain embodiments, the electronic device410transmits a single image that includes both the extended FoV744and the lookahead FoV752and video content between the two FoV's.

AlthoughFIGS.7A and7Billustrate the methods700and720, various changes may be made toFIGS.7A and7B. For example, while shown as a series of steps, various steps inFIGS.7A and7Bcould overlap, occur in parallel, or occur any number of times. The embodiments ofFIGS.7A and7Bare for illustration only. Other embodiments can be used without departing from the scope of this disclosure.

FIGS.8A and8Billustrate methods800and850, respectively, for split rendering of video content in accordance with an embodiment of this disclosure. In particular, the method800ofFIG.8Adescribes a server selecting portions of the video content while the method850ofFIG.8Bdescribes a client device receiving and displaying the portion of the received video content. The method800and850are described as implemented by any one of the client device106-116ofFIG.1, the server104ofFIG.1, the electronic device410, or the client device420, and can include internal components similar to that of server200ofFIG.2and the electronic device300ofFIG.3. However, the method800as shown inFIG.8Aand the method850shown inFIG.8Bcould be used with any other suitable electronic device and in any suitable system.

For ease of explanation, the method800is described as being performed by the electronic device410ofFIG.4Aand the method850is described as being performed by the client device420ofFIG.4A.

As shown in the method800ofFIG.8A, the electronic device410receives user data (step802). In certain embodiments, the user data is received from a client device, such as the client device420. The user data can include viewpoint data associated with a client device420. The viewpoint data indicates a portion of the 360° video that client device420is displaying for the user to view. The user data can also include multiple viewpoints, providing information for the electronic device (in step806) to determine the speed and direction of the client device. The user data can also include motion data. The motion data indicates motion of the client device420. For example, the motion data can indicate a velocity or speed of the client device420. The motion data can also indicate a direction of the motion. For example, if the client device420is an HMD (such as the HMD116ofFIG.1), then as the user moves their head, the client device420would also move.

In certain embodiments, the user data can correspond to one or more time instances. That is, the user data can correspond to a particular instance in time or multiple consecutive instances in time. For example, if the user data corresponds to a single instance in time, then the user data indicates a viewpoint and motion at a particular time. If the user data corresponds to multiple consecutive time instances, then the user data indicates multiple viewpoints over the user and the motion data indicating the speed and direction of client device420.

In step804, the electronic device410identifies a first portion of video content. The video content can be XR content. The first portion of the video content can correspond to a FoV of the client device420. In certain embodiments, the FoV is based on the viewpoint at a current time instance.

In step806, the electronic device410identifies a speed and direction of the client device420. The electronic device410can identify the speed and direction of the client device420based on the received motion data. That is, the motion data indicates a particular speed and direction of the client device at a current time instance. The electronic device410can identify the speed and direction of the client device420based on viewpoints from a set of consecutive time instances including the current time.

In step808, the electronic device410identifies a second portion of the video content. The electronic device410can use the identified speed and direction (from step806) to determine the size of the second portion of the video content. The second portion of the video content can extend from the first portion of the video content thereby expanding the first portion of the video content. In certain embodiments, the second portion of the video content extends asymmetrically. For example, the second portion of the video content extends asymmetrically along a direction as indicated in the motion data. That is, the second portion of the video content extends asymmetrically in a direction that corresponds to a direction that the client device420is moving. By extending the second portion of the video content asymmetrically along a particular direction, such the direction corresponding to the direction of the motion of the client device, enables electronic device410to provide more video content along a particular direction to the client device420.

In certain embodiments, the amount that the second portion of the video content is extended can be based on the speed (or velocity) of the head movement. A slow speed can indicate that the client device420is moving slowly and does not need a large extended video portion. In contrast, a high speed indicates that the client device420is moving fast and therefore the client device420could need an increase the extended video portion. Since the client device continues to move after sending the user data to the electronic device410and before receiving the video content from the electronic device410, the second portion of the video content enables the client device to display a portion of the received video content corresponding to the actual FoV, which could be different than the FoV that the electronic device410identified. As such, the amount that the second portion of the video content extends is based on the speed (velocity) of the client device. If the speed (velocity) of the client device420is slow (less than a threshold) indicates that the second portion of the video content can be reduced to decrease bandwidth. If the speed (velocity) of the client device420is high (larger than a threshold) indicates that the second portion of the video content should be increased as a safeguard to confirm that the client device420receives enough video content to be rendered and displayed for the user.

In step810, the electronic device410scales the second portion of the video content using a scaling factor. In certain embodiments, the electronic device410identifies the scaling factor based on the speed (velocity) of the client device420. For example, the electronic device410compares the speed to a threshold speed. The electronic device410can set the scaling factor to that of the first portion of the video content based on the speed being less than the threshold. That is, if the speed is less than a threshold, then the electronic device410uses the same scaling factor to scale the first portion of the video content and the second portion of the video content. Alternatively, the electronic device410can set the scaling factor to a different value. For example, if the speed is greater than the threshold, then the electronic device410uses a scaling factor that adds additional scaling to the second portion of the video content, beyond any scaling that is applied to the first portion of the video content. In certain embodiments, the amount of scaling cab be based on the speed (velocity) of the client device.

In certain embodiments, the electronic device410obtains content information. The electronic device410can then identify an event that occurs within the video content based on the obtained content information. The event corresponds to a portion of the video content that is separate from the first and second portions of the video content. For example, if the video content is a movie or a game, the event could correspond to an actor/character performing an action on a portion of the video where the user is not currently looking. The electronic device410can then determine whether to transmit a third portion of the video content to the client device420, based on the content information.

In certain embodiments, the user data includes preferences of a user of the client device. The preferences of the user can indicate certain events of the video content that the user would like to view. The electronic device410can then determine whether to transmit a third portion of the video content to the client device420, based on the content information and the preferences of the user. For example, if the content information indicates that an event will occur at Time, T, and the event is specified as a preference of the user, as indicated in the user data, then the electronic device410can determine to generate a third portion of the video content that corresponds to the event.

In step812, the electronic device410transmits the scaling factor as well as the first and second video portions of the video content to the client device420. In certain embodiments, the electronic device410transmits the scaling factor and the second portion of the video content to the client device, since the second portion of the video content includes the first portion of the video content.

In certain embodiments, the electronic device410renders the 3D XR content into two dimensional video frames. The electronic device410can also encode the first and second portions of the two dimensional video frames. Thereafter the electronic device410can transmit the two dimensional video frames to the client device420.

In certain embodiments, if the electronic device receives content information and determines to transmit the third portion of the video content corresponding to an event as indicated by the content information, the electronic device410transmits the first, second and third portions of the video content to the client device420.

In certain embodiments, if the content information indicates that an event will occur at Time, T, and the event is specified as a preference of the user, as indicated in the user data, then the electronic device410can generate a third portion of the video content. The electronic device410can then transmit the first, second and third portions of the video content to the client device420.

As shown in the method850ofFIG.8B, a client device420transmits user data to an electronic device410(step852). The user data can include motion data. The motion data can indicate a direction of motion, a speed of the motion, or both. The user data can also include a viewpoint, such as a viewing direction. In certain embodiments, the user data can include preferences that specifies certain events in the video content that the user wants to view.

In step854, the client device420receives a portion of the video content and a scaling factor. In certain embodiments, the portion of the video content is larger than a FoV of the client device. In certain embodiments, the portion of the video content can be asymmetrical.

In step856, the client device420scales the portion of the video content using the scaling factor. In certain embodiments, the client device420determines that the portion of the video content (received in step854) includes a first portion and a second portion. The first portion can correspond to a FoV corresponding to the time that the motion data and viewpoint were transmitted (step852). The second portion can extend from the first portion asymmetrically along a direction of the motion. The client device420then scales the second portion of the video content using the scaling factor.

In step858, the client device420selects a segment of the portion of the video that corresponds to the current FoV. The viewpoint that is transmitted to the client device in step852is a first viewpoint corresponding to a first time. The client device420identifies a second viewpoint after the portion of the video content is received at step854. The second viewpoint corresponds to a second time that is after the first time. The client device420selects the segment of the received video content that corresponds to the second viewpoint. Since the client device420can move positions from the first time and to the second time, the client device420selects a segment corresponding to the second time.

In step860, the client device420displays the segment of the video content on its display.

AlthoughFIGS.8A and8Billustrates one example of a methods800and850for split rendering video content, various changes may be made toFIGS.8A and8B. For example, while shown as a series of steps, various steps inFIGS.8A and8Bcould overlap, occur in parallel, or occur any number of times.

The above flowcharts illustrate example methods that can be implemented in accordance with the principles of the present disclosure and various changes could be made to the methods illustrated in the flowcharts herein. For example, while shown as a series of steps, various steps in each figure could overlap, occur in parallel, occur in a different order, or occur multiple times. In another example, steps may be omitted or replaced by other steps.

Although the figures illustrate different examples of user equipment, various changes may be made to the figures. For example, the user equipment can include any number of each component in any suitable arrangement. In general, the figures do not limit the scope of this disclosure to any particular configuration(s). Moreover, while figures illustrate operational environments in which various user equipment features disclosed in this patent document can be used, these features can be used in any other suitable system.

Although the present disclosure has been described with exemplary embodiments, various changes and modifications may be suggested to one skilled in the art. It is intended that the present disclosure encompass such changes and modifications as fall within the scope of the appended claims. None of the description in this application should be read as implying that any particular element, step, or function is an essential element that must be included in the claims scope. The scope of patented subject matter is defined by the claims.