Patent Description:
In a transmission process of a virtual reality (virtual reality, VR) video, one VR panoramic video may be split into one field of view (field of view, FoV) stream and one panoramic stream for transmission, where the FoV stream is video data of a current field of view of a user. For example, one <NUM> VR panoramic video may be split into one <NUM> FoV stream and one <NUM> panoramic stream.

Usually, a transmission procedure of a FoV stream of a VR video is shown in <FIG>. A specific process includes the following steps.

Step <NUM>. A terminal requests video data of a FoV1 from a server. In response to the request, the server starts to transmit the video data of the FoV1. After receiving the video data of the FoV1, the terminal decodes and displays the data.

FoV data requested by the terminal from the server is usually at a granularity of a second-level fragment (<NUM> to <NUM> frames).

Step <NUM>. The terminal perceives a change of a user's field of view (for example, the user turns the head), to be specific, detects that a FoV of the user changes from the FoV1 to a FoV2.

Step <NUM>. The terminal stops decoding and displaying the video data of the FoV1.

Step <NUM>. The terminal notifies the server to stop transmitting the video data of the FoV1. In response to the notification, the server stops transmitting the video data of the FoV1.

In addition, the terminal discards received video data of the FoV1, and the discarded FoV data of a previous field of view buffered by the terminal is redundant data.

Step <NUM>. The terminal requests video data of the FoV2 from the server. In response to the request, the server starts to transmit the video data of the FoV2. After receiving the video data of the FoV2, the terminal decodes and displays the data.

In this method, due to a specific delay in network transmission, when the FoV of the user changes, the server cannot receive in time a command sent by the terminal for stopping sending the video data of the FoV1. Within the delay, the server may continue to transmit the video data of the FoV1, and this results in generation of a large amount of redundant data. However, after the server receives signaling indicating that the terminal requests the video data of the FoV2, to compensate for time consumed by transmission of the redundant data of the FoV1 and a signaling transmission delay, the server increases a transmission bandwidth, to increase a transmission burst bandwidth, causing frame freezing or low-definition video display, and affecting user experience.

Document <CIT> discloses about a method, system and device for decreasing display lag in Cloud VR. The cloud server receives position and posture data from user's device. After then, an image that fits the user's view is returned by the server. This procedure improves the user's experience by speeding up and streamlining the VR experience by cutting down on the amount of time it takes to display the correct image.

Document <CIT> discloses about a system and process for managing virtual reality videos with the help of a graphics processing unit (GPU). Video data, more especially spherical image frames, are stored in memory by a video server. When a certain image frame is prepared, the server compresses the image frame and transforms it into a different kind of image using the GPU. By streamlining the processing and transmission of the videos, this system and process aims to improve user experience by increasing efficiency and smoothness during virtual reality video streaming.

Document <CIT> discloses about a technique for seamlessly switching between immersive video content on a virtual reality headset (HMD) and video content displayed on a standard TV The procedure involves receiving a signal from the TV containing a portion of the VR material, displaying it, and transmitting data about the location of the VR content in the virtual environment, such as its orientation or position. By comparing the VR programming with the standard TV content, this information is found. This signal can also include a timestamp, which specifies the time this VR material should be shown. Typically, this date can be delivered as a timed metadata stream. This technique facilitates a more seamless transition from standard TV to virtual reality material.

Document <CIT> discloses about a VR time synchronization system made up of a control module and a communication module. It facilitates to guarantee that the clocks of various virtual reality system components are synced. There are several slave devices and one master device in this system. The slave devices can modify their internal clocks to match the master device's clock by receiving a time synchronization signal from the master device such that the virtual reality system's devices are guaranteed to function with the same time reference. The system is appropriate for a range of virtual reality applications because it supports both wired and wireless connection.

Embodiments of this application provide a video transmission method, an apparatus, and a system, to reduce generation of redundant data in a VR video transmission process and reduce a transmission burst bandwidth. In addition, a motion-to-hires latency MTHR can be reduced. The invention is defined by the subject matter of the claims.

For ease of understanding of the technical solutions in embodiments of this application, the technologies related to this application are first briefly described as follows:.

<NUM> VR videos are equivalent only to videos on 240P televisions in users' views, which cannot meet basic experience requirements of users. To achieve good VR experience, <NUM> resolution is a basic condition for entry-level VR. However, mainstream VR terminals, especially all-in-one headsets, have only <NUM> decoding capabilities and cannot support <NUM> VR video decoding and display. In addition, a transmission bandwidth of <NUM> VR videos is generally over <NUM> million bits per second (million bits per second, Mbps), which exceeds transmission capabilities of mainstream home broadband and wireless fidelity (wireless-fidelity, Wi-Fi).

To enable a VR terminal to decode and display an <NUM> VR video, a mainstream processing manner in the industry is using a VR FoV technology.

Different from a conventional plane video, in a watching process, a user of a VR video can see only a video of <NUM>/<NUM> or even a smaller area. Therefore, in a process of watching the VR video, the user does not need to decode and display an entire <NUM> image, and only needs to decode and display an image in a current FoV of the user in real time. However, to ensure that the user can always see the image during a head turning process and ensure that the user is not dizzy during video watching, a <NUM> panoramic stream needs to be provided as a background image. According to this solution, one <NUM> VR panoramic video is split into one <NUM> FoV stream and one <NUM> panoramic stream. This solution may be understood as the VR FoV technology.

The VR FoV technology not only resolves the problem that VR terminals can decode only <NUM> videos, but also reduces a transmission bandwidth to less than <NUM> Mbps, meeting the transmission capability limitations of home broadband and Wi-Fi.

The MTHR is duration from a moment at which a terminal requests <NUM> FoV data corresponding to a latest FoV from a server to a moment at which the terminal device receives, decodes, and displays the <NUM> FoV data corresponding to the latest FoV.

The following describes the technical solutions in embodiments of this application with reference to the accompanying drawings in embodiments of this application. In the descriptions of this application, unless otherwise specified, "/" indicates an "or" relationship between associated objects. For example, A/B may indicate A or B. In this application, "and/or" describes only an association relationship between associated objects and indicates that three relationships may exist. For example, A and/or B may indicate the following three cases: Only A exists, both A and B exist, and only B exists, where A or B may be singular or plural. In addition, in the descriptions of this application, unless otherwise specified, "at least one" means one or more, and "a plurality of" means two or more. At least one of the following items (pieces) or a similar expression thereof refers to any combination of these items, including any combination of singular items (pieces) or plural items (pieces). For example, at least one item (piece) of a, b, or c may indicate: a, b, c, a and b, a and c, b and c, or a, b, and c, where a, b, and c may be singular or plural. In addition, to clearly describe the technical solutions in embodiments of this application, terms such as first and second are used in embodiments of this application to distinguish between same items or similar items that provide basically same functions or purposes. A person skilled in the art may understand that the terms such as "first" and "second" do not limit a quantity or an execution sequence, and that the terms such as "first" and "second" do not indicate a definite difference.

<FIG> shows a communication system <NUM> according to an embodiment of this application. The communication system <NUM> includes a server <NUM> and a terminal <NUM>.

There are two links between the server <NUM> and the terminal <NUM>: a first link and a second link. The first link is for transmitting field of view information, and the second link is for transmitting image data corresponding to the field of view information.

In this embodiment of this application, the terminal <NUM> sends first field of view information to the server <NUM> through the first link based on a first period. Correspondingly, the server <NUM> receives the first field of view information from the terminal through the first link based on the first period. After determining, based on a second period, that the first field of view information is latest field of view information, the server <NUM> obtains data of a first frame of image corresponding to the first field of view information, and sends the data of the first frame of image to the terminal <NUM> through the second link. Correspondingly, the terminal <NUM> receives the data of the first frame of image from the server <NUM> through the second link based on the second period. Detailed descriptions of the solution are to be described in subsequent embodiments, and are not described herein.

It should be noted that in this embodiment of this application, the "image" may also be referred to as a "FoV frame" or a "FoV image frame", and the names may be replaced with each other. This is not specifically limited in this embodiment of this application.

In the solution in this embodiment of this application, the server obtains, based on the second period, the latest field of view information reported by the terminal and FoV image data corresponding to the latest field of view information, that is, time for generating and sending the FoV image data on the server side is independently controlled by the server. In addition, each time the server generates and sends data of a frame of image corresponding to latest field of view information, a transmission granularity can be reduced to an image frame level in this embodiment, to more precisely control processing and sending time of each frame of image, reduce generation and transmission of redundant data, and further reduce a transmission burst bandwidth. In addition, the terminal reports the field of view information through the first link, and the server sends the image data through the second link. Therefore, an independent link is for reporting the field of view information, and sending and receiving of the image data are not affected. When image data corresponding to new field of view information is transmitted, a transmission link does not need to be re-established, to reduce an MTHR.

Optionally, the server <NUM> in this embodiment of this application may be a VR media server. The server <NUM> may be deployed in a mobile edge computing (mobile edge computing, MEC) environment. MEC is a technology for deeply integrating a mobile network and an internet service based on a 5th generation (5th generation, <NUM>) evolution architecture. The 3rd Generation Partnership Project (3rd Generation Partnership Project, 3GPP) initiated MEC-related projects in both a radio access network (radio access network, RAN) <NUM> workgroup and a service and architecture (service and architecture, SA) <NUM> workgroup.

Optionally, the terminal <NUM> in this embodiment of this application may be a device configured to display a VR video, for example, a wearable device, a VR terminal, an augmented reality (augmented reality, AR) terminal, a wireless terminal in industrial control (industrial control), a wireless terminal in self driving (self driving), a wireless terminal in remote medical (remote medical), a wireless terminal in a smart grid (smart grid), a wireless terminal in transportation safety (transportation safety), a wireless terminal in a smart city (smart city), or a wireless terminal in smart home (smart home). The terminal may be mobile or fixed.

Optionally, the server <NUM> and the terminal <NUM> in this embodiment of this application may also be referred to as communication apparatuses, and each may be a general-purpose device or a dedicated device. This is not specifically limited in this embodiment of this application.

Optionally, in this embodiment of this application, the server <NUM> or the terminal <NUM> may be implemented by a communication device (or communication apparatus) <NUM> in <FIG>.

<FIG> is a schematic diagram of a structure of the communication device <NUM> according to an embodiment of this application. The communication device <NUM> includes one or more processors <NUM> and at least one communication interface (where <FIG> is merely an example in which a communication interface <NUM> and one processor <NUM> are included for description), and optionally may further include a memory <NUM> and a communication bus <NUM>.

Optionally, the processor <NUM>, the communication interface <NUM>, or the memory <NUM> may be coupled together (not shown in <FIG>), or as shown in <FIG>, may be connected together through the communication bus <NUM>.

The processor <NUM> may be a general-purpose central processing unit (central processing unit, CPU), a microprocessor, an application-specific integrated circuit (application-specific integrated circuit, ASIC), or one or more integrated circuits configured to control program execution of the solutions of this application.

The communication bus <NUM> may be a peripheral component interconnect (peripheral component interconnect, PCI) bus, an extended industry standard architecture (extended industry standard architecture, EISA) bus, or the like. The bus may be classified into an address bus, a data bus, a control bus, and the like. For ease of representation, only one thick dashed line is used to represent the bus in <FIG>, but this does not indicate that there is only one bus or only one type of bus. The communication bus <NUM> may be configured to connect different components in the communication device <NUM>, so that the different components can communicate with each other.

The communication interface <NUM> may be a transceiver module, configured to communicate with another device or a communication network, for example, an Ethernet, a radio access network (radio access network, RAN), or a wireless local area network (wireless local area network, WLAN). For example, the transceiver module may be an apparatus such as a transceiver or a transceiver. Optionally, the communication interface <NUM> may alternatively be a transceiver circuit located inside the processor <NUM>, and is configured to implement signal input and signal output of the processor.

The memory <NUM> may be an apparatus having a storage function. For example, the memory may be a read-only memory (read-only memory, ROM), another type of static storage device that can store static information and instructions, a random access memory (random access memory, RAM), or another type of dynamic storage device that can store information and instructions, or may be an electrically erasable programmable read-only memory (electrically erasable programmable read-only memory, EEPROM), a compact disc read-only memory (compact disc read-only memory, CD-ROM) or another compact disc storage, an optical disc storage (including a compact disc, a laser disc, an optical disc, a digital versatile disc, a Blu-ray disc, or the like), a magnetic disk storage medium or another magnetic storage device, or any other medium that can be configured to carry or store expected program code in a form of instructions or a data structure and that can be accessed by a computer. However, the memory is not limited thereto. The memory may exist independently, and connect to the processor through the communication line <NUM>. The memory may alternatively be integrated with the processor.

The memory <NUM> is configured to store computer instructions for performing the solutions of this application, and the processor <NUM> controls execution of the computer instructions. The processor <NUM> is configured to execute the computer instructions stored in the memory <NUM>, to implement the method provided in embodiments of this application.

Alternatively, optionally, in this embodiment of this application, the processor <NUM> may perform processing-related functions in a method provided in the following embodiments of this application, and the communication interface <NUM> is responsible for communication with another device or a communication network. This is not specifically limited in this embodiment of this application.

Optionally, the computer instructions in this embodiment of this application may also be referred to as application program code. This is not specifically limited in this embodiment of this application.

During specific implementation, in an embodiment, the processor <NUM> may include one or more CPUs, for example, a CPU0 and a CPU1 in <FIG>.

During specific implementation, in an embodiment, the communication device <NUM> may include a plurality of processors, for example, the processor <NUM> and a processor <NUM> in <FIG>. Each of the processors may be a single-core (single-CPU) processor, or may be a multi-core (multi-CPU) processor. The processor herein may be one or more devices, circuits, and/or processing cores configured to process data (for example, computer program instructions).

It may be understood that the structure shown in <FIG> constitute no specific limitation on the communication apparatus. For example, in some other embodiments of this application, the communication apparatus may include more or fewer components than those shown in the figure, some components may be combined, some components may be split, or the components may be differently arranged. The components shown in the figure may be implemented by hardware, software, or a combination of software and hardware.

Optionally, <FIG> is a structural block diagram of a server <NUM> and a terminal <NUM> according to an embodiment of this application. The structural block diagram may be understood as division of the server <NUM> and the terminal <NUM> from a perspective of logical functions.

The terminal <NUM> includes one or more of the following modules: a field of view perception module, a field of view information reporting module, a terminal clock module, a media data stream-receiving module, a video decoding module, and a video rendering module.

It may be understood that modules of the terminal <NUM> may communicate with each other. <FIG> is merely an example of a schematic diagram of communication between modules of the terminal <NUM>, and does not constitute any limitation.

It should be noted that a manner of communication between modules of the server <NUM> and the modules of the terminal <NUM> is not limited in this embodiment of this application. <FIG> is merely an example of a schematic diagram of communication between the modules of the server <NUM> and the modules of the terminal <NUM>, and there may be another communication manner.

Optionally, the field of view perception module is mainly configured to monitor a field of view of a user. The field of view perception module may monitor a change of a field of view caused by a head turn or another displacement of the user, and transfer the change to another module such as the field of view information reporting module of the terminal.

The field of view information reporting module is mainly configured to: establish and maintain an independent long-term transmission channel (namely, a first link) with the server, and periodically report latest field of view information of the user to the server through the transmission channel based on a first period.

The terminal clock module is mainly configured to provide unified system clock and timer services for each module of the terminal, and is configured to coordinate a processing time sequence between the modules. In addition, the terminal clock module is further configured to perform clock synchronization with the server.

The media data stream-receiving module is mainly configured to: establish and maintain an independent media data transmission channel (that is, a second link) with the server, and receive image data from the server through the channel based on a second period.

The video decoding module is mainly configured to decode the received image data.

The video rendering module is mainly configured to render the received image data, combine the image data with a background image, and display a combined image data.

For the server <NUM>:
The server <NUM> includes one or more of the following modules: a field of view information receiving module, a field of view information processing module, a server clock module, a FoV frame buffering module, a FoV frame streaming module, and a media data stream-sending module.

It may be understood that the modules of the server <NUM> may communicate with each other. <FIG> is merely an example of a schematic diagram of communication between the modules of the server <NUM>, and does not constitute any limitation.

Optionally, the field of view information receiving module is mainly configured to: establish and maintain an independent long-term transmission channel (namely, the first link) with the terminal, receive the field of view information reported by the terminal periodically through the transmission channel based on the first period, and provide the field of view information to the field of view information processing module.

The field of view information processing module is mainly configured to process the field of view information reported by the terminal, and convert the field of view information into a unified coordinate description on the server side.

Optionally, the field of view information processing module may further include a field of view information processing logic engine module and a field of view information buffering module. The field of view information processing logic engine module is configured to process the field of view information reported by the terminal, and the field of view information buffering module is configured to buffer processed field of view information.

The server clock module is mainly configured to provide unified system clock and timer services for each module of the server, and is configured to coordinate a processing time sequence between the modules. In addition, the server clock module is further configured to receive a synchronization message from the terminal.

The FoV frame buffering module is mainly configured to buffer processed FoV frame data.

The FoV frame streaming module mainly selects a FoV frame corresponding to the field of view information based on a frame time sequence and a server clock, and encapsulates the FoV frame.

The media data stream-sending module is mainly configured to: establish and maintain an independent media data transmission channel (that is, the second link) with the terminal, and send image data to the terminal through the channel based on the second period.

It may be understood that the structure shown in <FIG> does not constitute a specific limitation on the terminal or the server. For example, in some other embodiments of this application, the terminal or the server may include more or fewer modules than those shown in the figure, some modules may be combined, some modules may be split, or the modules may be differently arranged. The modules shown in the figure may be implemented by hardware, software, or a combination of software and hardware.

It may be understood that names of the modules of the terminal or the server shown in <FIG> are merely examples, and other names may also be used during specific implementation. This is not limited.

The following describes in detail the video transmission method provided in embodiments of this application with reference to <FIG>.

It may be understood that actions, terms, and the like in embodiments of this application may be mutually referenced. This is not limited. In embodiments of this application, names of messages exchanged between devices, names of parameters in the messages, or the like are merely examples. Another name may alternatively be used during specific implementation. This is not limited.

In addition, in embodiments of this application, the terminal or the server may perform some or all steps in embodiments of this application. These steps or operations are merely examples. In embodiments of this application, other operations or variations of various operations may be further performed. In addition, the steps may be performed in a sequence different from a sequence presented in embodiments of this application, and not all operations in embodiments of this application need to be performed.

Based on the structural block diagram of the terminal or the server shown in <FIG>, <FIG> and <FIG> show a video transmission method according to an embodiment of this application. The transmission method includes the following steps.

A terminal establishes a first link and a second link to a server.

The first link is for transmitting field of view information, and the second link is for transmitting a FoV frame. The first link and the second link are different links and are independent of each other. That is, data transmission on the first link and the second link does not affect each other.

Optionally, after step S500 is completed, it may be considered that the first link is established between a field of view information reporting module of the terminal and a field of view information receiving module of the server, and the second link is established between a media data stream-receiving module of the terminal and a media data stream-sending module of the server.

Optionally, step S500 may be performed when the terminal device starts to play a video.

The terminal sends first information to the server. Correspondingly, the server receives the first information from the terminal.

The first information indicates a first reference moment for sending image data to the terminal, and the first reference moment is a moment at which the server sends the image data to the terminal for the first time.

Optionally, the terminal may send the first information to the terminal after the terminal completes preparation work such as establishment of the first link and the second link and can start to transmit data to the server.

Optionally, the first information may include the first reference moment.

Optionally, the "first time" in this embodiment of this application may be understood as the first time for a video, instead of the first time after power-on of the terminal. For example, after the terminal is powered on, the terminal first plays a video <NUM> and then plays a video <NUM>. In this case, for the video <NUM>, the first reference moment is a moment at which the server sends image data of the video <NUM> to the terminal for the first time; and for the video <NUM>, the first reference moment is a moment at which the server sends image data of the video <NUM> to the terminal for the first time.

Optionally, that the terminal sends the first information to the server may be understood as that the terminal initiates one time of clock synchronization to the server. After receiving the first information, the server may establish a clock instance for the terminal, and subsequently perform time sequence control and processing based on the clock instance. To be specific, in the following embodiment, a server clock module of the server may perform time sequence control based on the clock instance and a second period.

Afield of view perception module of the terminal perceives a current field of view of the terminal based on a first period.

Optionally, the "current field of view of the terminal" may also be understood as a "current field of view of a user who uses the terminal". This is not specifically limited in this embodiment of this application.

Optionally, a terminal clock module may activate the field of view perception module periodically based on the first period, so that the field of view perception module perceives the current field of view of the terminal based on the first period. In other words, the terminal may perceive the current field of view of the terminal based on a clock of the terminal and the first period.

Optionally, a start moment of the <NUM>st first period may be the first reference moment. In other words, a moment at which the terminal perceives the current field of view of the terminal for the first time may be the first reference moment. Then, the terminal periodically perceives the current field of view of the terminal based on the first reference moment and the first period.

Optionally, after perceiving the current field of view of the terminal, the field of view perception module of the terminal generates field of view information of the current field of view, that is, the field of view information is for describing the current field of view perceived by the field of view perception module.

The field of view perception module of the terminal sends the field of view information to the field of view information reporting module. Correspondingly, the field of view information reporting module of the terminal receives the field of view information from the field of view perception module.

It may be understood that the field of view perception module of the terminal may send the field of view information to the field of view information reporting module after generating the field of view information of the current field of view.

The field of view information reporting module of the terminal sends the field of view information to the field of view information receiving module of the server through the first link. Correspondingly, the field of view information receiving module of the server receives the field of view information from the field of view information reporting module of the terminal through the first link.

It may be understood that, after receiving the field of view information from the field of view perception module, the field of view information reporting module of the terminal may report the field of view information to the field of view information receiving module of the server through the first link. Therefore, a period for the terminal to report the field of view information to the server may be considered as the first period. In other words, the terminal may send first field of view information to the server based on the clock of the terminal and the first period.

In this embodiment of this application, an example in which the field of view information reported by the terminal to the server in the first period closest to a current moment is the first field of view information is used for description.

The field of view information receiving module of the server sends the field of view information to a field of view information processing module. Correspondingly, the field of view information processing module of the server receives the field of view information from the field of view information receiving module.

Optionally, after receiving the field of view information from the terminal, the field of view information receiving module of the server may send the field of view information to the field of view information processing module of the server.

The field of view information processing module of the server processes the field of view information, and buffers processed field of view information.

Optionally, step S506 may include: Afield of view information processing logic engine module included in the field of view information processing module of the server processes the field of view information, for example, converting the field of view information into unified field of view coordinates on the server side, where the field of view coordinates are the processed field of view information; and a field of view information buffering module included in the field of view information processing module of the server buffers the processed field of view information.

In this embodiment of this application, it may be considered that the pre-processed field of view information and the processed field of view information is same information. The following described field of view information is specifically the pre-processed field of view information or the processed field of view information, and may be determined based on a scenario in which the field of view information appears. Details are not described in this embodiment of this application.

A FoV frame streaming module of the server obtains latest field of view information from the field of view information processing module based on the second period.

Optionally, the second period is determined based on a frame rate of a video, to be specific, the second period is a display interval between two adjacent frames of images at the frame rate. The video is a video that the terminal starts to play. For example, a frame rate at which a quantity of frames transmitted per second (frames per second, FPS) is <NUM> frames (that is, <NUM> FPS) is used as an example, and the second period is <NUM> milliseconds (ms).

Optionally, the second period and the first period may be the same, or may be different. When the two periods are different, a relationship between sizes of the second period and the first period is not specifically limited in this embodiment of this application.

Optionally, the server clock module of the server may activate the FoV frame streaming module periodically based on the second period, so that the FoV frame streaming module obtains the latest field of view information based on the second period.

Optionally, the FoV frame streaming module may determine field of view information buffered by the field of view information processing module at a moment closest to the current moment as the latest field of view information.

In this embodiment of this application, because the field of view information reported by the terminal at a moment closest to the current moment is the first field of view information, the FoV frame streaming module of the server determines, based on the second period, that the latest field of view information buffered by the field of view information processing module is the first field of view information, that is, the server may determine, based on the second period, that the first field of view information is the latest field of view information.

Optionally, a start moment of the <NUM>st second period may be the first reference moment. In other words, a moment at which the server obtains the latest field of view information for the first time may be the first reference moment. Then, latest field of view information is obtained based on the first reference moment and the second period. To be specific, that the server determines, based on the second period, that the first field of view information is the latest field of view information may include: The server determines, based on a clock of the server and the second period, that the first field of view information is the latest field of view information.

The FoV frame streaming module of the server obtains, from a FoV frame buffer, data of a first frame of image corresponding to the first field of view information.

That is, the FoV frame streaming module of the server obtains, from the FoV frame buffer, data of a frame of image corresponding to latest field of view information.

Optionally, after obtaining data of the frame of image corresponding to the latest field of view information, the FoV frame streaming module of the server may re-encapsulate the data of the frame of image, and encapsulate the data in a FoV frame stream.

It may be understood that after obtaining the latest field of view information, the FoV frame streaming module of the server may obtain the data of the frame of image corresponding to the latest field of view information. Therefore, it may be considered that a period in which the server obtains data of a frame of image corresponding to latest field of view information is also the second period.

The FoV frame streaming module of the server sends the data of the first frame of image to the media data stream-sending module. Correspondingly, the media data stream-sending module of the server receives the data of the first frame of image from the FoV frame streaming module.

That is, the FoV frame streaming module of the server sends, to the media data stream-sending module, the data of the frame of image corresponding to the latest field of view information.

It may be understood that, after obtaining the data of the frame of image corresponding to the latest field of view information, the FoV frame streaming module of the server may send the data of the image to the media data stream-sending module. Therefore, it may be considered that a period in which the media data stream-sending module of the server receives data of a frame of image corresponding to latest field of view information is also the second period. The media data stream-sending module of the server sends the data of the first frame of image to the media data stream-receiving module of the terminal through the second link. Correspondingly, the media data stream-receiving module of the terminal receives the data of the first frame of image from the media data stream-sending module of the server through the second link.

That is, the media data stream-sending module of the server sends, to the media data stream-receiving module of the terminal, the data of the frame of image corresponding to the latest field of view information.

Optionally, after receiving the data of the first frame of image, the media data stream-receiving module of the terminal may buffer the first frame of image, and subsequently send, according to a processing time sequence of the terminal, the first frame of image to a video decoding module for decoding. After the decoding, the video decoding module transmits the first frame of image to a video rendering module for rendering, to finally display the first frame of image.

Optionally, after receiving the data of the first frame of image, the media data stream-sending module of the server may send the data of the first frame of image to the terminal. Therefore, it may be considered that a period in which the server sends data of a frame of image corresponding to latest field of view information is also the second period. Alternatively, the server clock module of the server may periodically activate the media data stream-sending module based on the second period, so that the media data stream-sending module sends data of a frame of image corresponding to latest field of view information to the terminal based on the second period.

Accordingly, because the server sends data of a frame of image based on the second period, it may be considered that the terminal receives the data of the frame of image based on the second period.

In conclusion, in this embodiment of this application, the terminal reports the field of view information to the server based on the first period. After receiving the field of view information, the server does not immediately obtain an image corresponding to the field of view information, but obtains, based on the second period, latest field of view information considered by the server, and then sends data of a frame of image corresponding to latest field of view information to the terminal. Therefore, in general, in this embodiment of this application, the server sends video data to the terminal frame by frame.

Based on the solution in this embodiment of this application, the server obtains, based on the second period, the latest field of view information reported by the terminal and the data of the frame of image corresponding to the latest field of view information, that is, time for generating and sending image data on the server side is independently controlled by the server. In addition, each time the server generates and sends data of a frame of image corresponding to latest field of view information, a transmission granularity can be reduced to an image frame level, to more precisely control processing and sending time of each frame of image, reduce generation and transmission of redundant data, and further reduce a transmission burst bandwidth. In addition, the terminal reports the field of view information through the first link, and the server sends the image data through the second link. Therefore, an independent link is for reporting the field of view information, and sending and receiving of the image data are not affected. When image data corresponding to new field of view information is transmitted, a transmission link does not need to be re-established, to reduce an MTHR.

It may be understood that when a structure of the terminal or the server is not the structure shown in <FIG>, the video transmission method provided in this embodiment of this application is still applicable. To be specific, the terminal can still implement functions/implementation processes implemented by the modules of the terminal shown in <FIG> and <FIG>, and the server can still implement functions/implementation processes implemented by the modules of the server shown in <FIG> and <FIG>.

The following describes, by using a specific example, application of the video transmission method shown in <FIG> and <FIG>.

For example, the terminal plays a video <NUM>, and the first reference moment is <NUM> o'clock. Assuming that the first period is <NUM>, and a frame rate of the video <NUM> is <NUM> FPS, that is, the second period is <NUM>, processing time sequences of the terminal and the server may be shown in <FIG>. To be specific, the terminal reports field of view information to the server every <NUM> starting from <NUM> o'clock, and the server obtains latest field of view information and data of a frame of image corresponding to the latest field of view information every <NUM> starting from <NUM> o'clock, and sends the latest field of view information and the data to the terminal.

As shown in <FIG>, latest field of view information obtained by the server for the first time is field of view information <NUM>, latest field of view information obtained for the second time is field of view information <NUM>, and latest field of view information obtained for the third time is field of view information <NUM>. In other words, the server sends, to the terminal based on the second period, data of frames of images corresponding to the field of view information <NUM>, the field of view information <NUM>, and the field of view information <NUM> respectively.

It may be understood that, in this scenario, when the server sends, to the terminal, the frame of image corresponding to the field of view information <NUM>, even if the server sends, to the terminal <NUM> after sending the frame of image corresponding to the field of view information <NUM>, the frame of image corresponding to the field of view information <NUM>, because a display interval between the two frames of images is <NUM>, the terminal cannot display the frame of image corresponding to the field of view information <NUM>. Therefore, frame skipping usually does not occur when the server sends, based on the second period, a frame of image corresponding to latest field of view information. In this way, user experience is not affected.

Alternatively, for example, the terminal plays a video <NUM>, and the first reference moment is <NUM> o'clock. Assuming that the first period is <NUM>, and a frame rate of the video <NUM> is <NUM> FPS, that is, the second period is <NUM>, processing time sequences of the terminal and the server may be shown in <FIG>. To be specific, the terminal reports field of view information to the server every <NUM> starting from <NUM> o'clock, and the server obtains latest field of view information and data of a frame of image corresponding to the latest field of view information every <NUM> starting from <NUM> o'clock, and sends the latest field of view information and the data to the terminal.

As shown in <FIG>, latest field of view information obtained by the server for the first time is field of view information <NUM>, latest field of view information obtained for the second time is the field of view information <NUM>, latest field of view information obtained for the third time is field of view information <NUM>, latest field of view information obtained for the fourth time is field of view information <NUM>, latest field of view information obtained for the fifth time is field of view information <NUM>, and latest field of view information obtained for the sixth time is field of view information <NUM>. In other words, the server separately sends, to the terminal at different moments based on the second period, data of a frame of image corresponding to the field of view information <NUM> twice and data of a frame of image corresponding to each of the field of view information <NUM>, the field of view information <NUM>, the field of view information <NUM>, and the field of view information <NUM>.

It may be understood that, in this scenario, after receiving field of view information even if the server does not immediately obtain and send a frame of image corresponding to the field of view information, because the server performs processing at a granularity of a frame, generation of redundant data is greatly reduced, and time for transmitting image data corresponding to old field of view information in a network is short. Therefore, time required for transmitting image data corresponding to new field of view information is short. Overall, an MTHR can still be reduced in the solution in this embodiment of this application.

It may be understood that the scenario shown in <FIG> is merely intended to describe the solutions in embodiments of this application more clearly. In actual application, to ensure terminal sensitivity and user experience, the first period is not excessively long, and is generally from <NUM> to <NUM>.

Optionally, in an implementation scenario of this embodiment of this application, as shown in <FIG> and <FIG>, the video transmission method may further include the following step.

The terminal sends second information to the server. Correspondingly, the server receives the second information from the terminal.

Optionally, the terminal may periodically determine a quantity of frames of an image buffered by the terminal, and send the second information to the server based on the quantity of frames of the image.

In a possible implementation, when the quantity of frames of the image buffered by the terminal is greater than or equal to a first threshold, the terminal sends the second information to the server, where the second information indicates to send data of a second frame of image at a second reference moment, the second reference moment is later than a first moment, the first moment is a moment at which the second frame of image is sent based on the second period by using the first reference moment as a start moment, and the second frame of image is a next frame of image to be received by the terminal. Alternatively, the first moment is a moment at which the second frame of image is sent based on the second period that uses the first reference moment as a start moment. That is, the server delays sending the second frame of image.

Optionally, reasons why the quantity of frames of the image buffered by the terminal is greater than or equal to the first threshold may be that network quality is good and a transmission delay is low. In this case, if the server sends image data based on the second period by using the first reference moment as the start moment, the terminal may buffer excessive image data. In this case, the terminal may send the second information to the server, to indicate the server to send the second frame of image at the second reference moment, where the second reference moment is later than the first moment.

For example, if the first threshold is <NUM>, image frames buffered by the terminal are the <NUM>th frame and the <NUM>th frame, and the first reference moment is <NUM> o'clock, a frame number of the second frame of image is <NUM>. It is assumed that the first reference moment is used as a start moment, and a moment (that is, the first moment) at which the second frame of image is sent based on the second period is <NUM> o'clock. In this case, the second reference moment is later than <NUM> o'clock. For example, the second reference moment may be <NUM>:<NUM>.

In another possible implementation, when the quantity of frames of the image buffered by the terminal is less than or equal to a second threshold, the terminal sends the second information to the server, where the second information indicates to send data of a second frame of image at a second reference moment, the second reference moment is earlier than a first moment, the first moment is a moment at which the second frame of image is sent based on the second period by using the first reference moment as a start moment, and the second frame of image is a next frame of image to be received by the terminal. Alternatively, the first moment is a moment at which the second frame of image is sent based on the second period that uses the first reference moment as a start moment. That is, the server sends the second frame of image in advance.

Optionally, reasons why the quantity of frames of the image buffered by the terminal is less than or equal to the second threshold may be that network quality is poor and a transmission delay is high. In this case, if the server sends image data based on the second period by using the first reference moment as the start moment, frame freezing or erratic display may occur when the terminal plays a video, affecting user experience. In this case, the terminal may send the second information to the server, to indicate the server to send the second frame of image at the second reference moment, where the second reference moment is earlier than the first moment.

For example, if the second threshold is <NUM>, an image frame buffered by the terminal is the <NUM>th frame, and the first reference moment is <NUM> o'clock, a frame number of the second frame of image is <NUM>. It is assumed that the first reference moment is used as a start moment, and a moment (that is, the first moment) at which the second frame of image is sent based on the second period is <NUM> o'clock. In this case, the second reference moment is earlier than <NUM> o'clock. For example, the second reference moment may be <NUM>:<NUM>.

Optionally, in the foregoing two implementations, the second information may include the frame number of the second frame of image. In this case, the server may determine a moment at which the second information is received as the second reference moment, and obtain and send the data of the second frame of image at the second reference moment. Alternatively, the second information may include the second reference moment. In this case, the server may determine a next frame of image to be received by the terminal as a next frame of image to be sent by the server, and obtain and send the data of the second frame of image at the second reference moment. Alternatively, the second information may include the second reference moment and the frame number of the second frame of image. In this case, the server may send the data of the second frame of image at the second reference moment indicated by the second information.

Optionally, in the foregoing two implementations, after sending the data of the second frame of image at the second reference moment, the server may continue to obtain and send image data based on the second period by using the second reference moment as a start moment.

Optionally, that the terminal sends the second information to the server may be understood as that the terminal initiates another time of clock synchronization to the server.

Based on the solution, the terminal can dynamically adjust, in real time by perceiving a change of the quantity of image frames buffered by the terminal, time for sending image data by the server. On one hand, the terminal may be prevented from buffering an excessively large amount of image data, and a redundant data amount in a transmission process can be reduced, to reduce a transmission burst bandwidth when a user's field of view changes (for example, the user turns the head). On the other hand, this can effectively alleviate a problem of transmission instability caused by a network condition change, especially a transmission delay jitter, improve transmission stability, and improve user experience.

In embodiments of this application, unless otherwise stated or there is a logic conflict, terms and/or descriptions between different embodiments are consistent and may be mutually referenced, and technical features in different embodiments may be combined based on an internal logical relationship thereof, to form a new embodiment.

It may be understood that, in the foregoing embodiments, methods and/or steps implemented by the server may also be implemented by a component (for example, a chip or a circuit) that can be used in the server, and methods and/or steps implemented by the terminal may also be implemented by a component (for example, a chip or a circuit) that can be used in the terminal.

The foregoing mainly describes, from a perspective of interaction between devices, the solutions provided in embodiments of this application. Correspondingly, embodiments of this application further provide a communication apparatus, and the communication apparatus is configured to implement the foregoing various methods. The communication apparatus may be the terminal in the foregoing method embodiments, an apparatus including the foregoing terminal, or a component that can be used in the terminal. Alternatively, the communication apparatus may be the server in the foregoing method embodiments, an apparatus including the foregoing server, or a component that can be used in the server. It may be understood that, to implement the foregoing functions, the communication apparatus includes a hardware structure and/or a software module for performing a corresponding function. A person skilled in the art should easily be aware that, in combination with units and algorithm steps of the examples described in embodiments disclosed in this specification, this application may be implemented by hardware or a combination of hardware and computer software. Whether a function is performed by hardware or hardware driven by computer software depends on particular applications and design constraints of the technical solutions.

In embodiments of this application, the communication apparatus may be divided into function modules based on the foregoing method embodiments. For example, each function module may be obtained through division based on each corresponding function, or two or more functions may be integrated into one processing module. The integrated module may be implemented in a form of hardware, or may be implemented in a form of a software functional module. It should be noted that, in embodiments of this application, module division is an example, and is merely a logical function division. In actual implementation, another division manner may be used.

For example, the communication apparatus is the server in the foregoing method embodiments. <FIG> is a schematic diagram of a structure of a server <NUM>. The server <NUM> includes a processing module <NUM> and a transceiver module <NUM>. The transceiver module <NUM> may also be referred to as a transceiver unit for implementing a sending and/or receiving function, and may be, for example, a transceiver circuit, a transceiver machine, a transceiver, or a communication interface.

The transceiver module <NUM> is configured to receive first field of view information from a terminal through a first link based on a first period. The processing module <NUM> is configured to determine, based on a second period, that the first field of view information is latest field of view information. The processing module <NUM> is further configured to obtain data of a first frame of image corresponding to the first field of view information. The transceiver module <NUM> is further configured to send the data of the first frame of image to the terminal through a second link.

Optionally, that the processing module <NUM> is configured to determine, based on a second period, that the first field of view information is latest field of view information may include: The processing module <NUM> is configured to determine, based on a clock of the server and the second period, that the first field of view information is the latest field of view information.

Optionally, the transceiver module <NUM> is further configured to receive first information from the terminal, where the first information indicates a first reference moment for sending image data to the terminal, and the first reference moment is a moment at which the server sends the image data to the terminal for the first time.

Optionally, the transceiver module <NUM> is further configured to receive second information from the terminal, where the second information indicates to send data of a second frame of image at a second reference moment, the second frame of image is a next frame of image to be received by the terminal, the second reference moment is later than a first moment or the second reference moment is earlier than the first moment, and the first moment is a moment at which the second frame of image is sent based on the second period that uses the first reference moment as a start moment.

All related content of the steps in the foregoing method embodiments may be cited in function descriptions of the corresponding functional modules.

Optionally, the server <NUM> may further include a storage module (not shown in <FIG>), configured to store data and/or instructions. The processing module <NUM> may read the data or the instructions in the storage module, to implement the methods corresponding to the foregoing embodiments.

It may be understood that the foregoing modules may be independently disposed, or may be integrated. This is not limited in this embodiment of this application.

In a possible implementation, the server <NUM> is presented in an integrated form of function modules obtained through division. The "module" herein may be an ASIC, a circuit, a processor that executes one or more software or firmware programs, a memory, an integrated logic circuit, and/or another component capable of providing the foregoing functions. In a simple embodiment, a person skilled in the art can figure out that the form of the communication device <NUM> shown in <FIG> may be used for the server <NUM>.

For example, the processor <NUM> in the communication device <NUM> shown in <FIG> may invoke the computer-executable instructions stored in the memory <NUM>, to enable the communication device <NUM> to perform the video transmission methods in the foregoing method embodiments.

Specifically, functions/implementation processes of the processing module <NUM> and the transceiver module <NUM> in <FIG> may be implemented by the processor <NUM> in the communication device <NUM> shown in <FIG> by invoking the computer-executable instructions stored in the memory <NUM>. Alternatively, a function/an implementation process of the processing module <NUM> in <FIG> may be implemented by the processor <NUM> in the communication device <NUM> shown in <FIG> by invoking the computer-executable instruction stored in the memory <NUM>, and a function/an implementation process of the transceiver module <NUM> in <FIG> may be implemented by using the communication interface <NUM> in the communication device <NUM> shown in <FIG>.

Because the server <NUM> provided in this embodiment may perform the foregoing video transmission method, for a technical effect that can be achieved by the server, refer to the foregoing method embodiments.

Alternatively, for example, the communication apparatus is the terminal the foregoing method embodiments. <FIG> is a schematic diagram of a structure of a terminal <NUM>. The terminal <NUM> includes a sending module <NUM> and a receiving module <NUM>. The sending module <NUM> may also be referred to as a sending unit for implementing a sending function, and the receiving module <NUM> may also be referred to as a receiving unit for implementing a receiving function. The sending module <NUM> and the receiving module <NUM> each may be, for example, a transceiver circuit, a transceiver machine, a transceiver, or a communication interface.

The sending module <NUM> is configured to send first field of view information to a server through a first link based on a first period. The receiving module <NUM> is configured to receive, from the server through a second link based on a second period, data of a first frame of image that corresponds to the first field of view information.

Optionally, that the sending module <NUM> is configured to send first field of view information to a server through a first link based on a first period may include: The sending module <NUM> is configured to send the first field of view information to the server through the first link based on a clock of the terminal and the first period.

Optionally, the sending module <NUM> is further configured to send first information to the server, where the first information indicates a first reference moment for sending image data to the terminal, and the first reference moment is a moment at which the server sends the image data to the terminal for the first time.

Optionally, the sending module <NUM> is further configured to send second information to the server when a quantity of frames of an image buffered by the terminal is greater than or equal to a first threshold, where the second information indicates to send data of a second frame of image at a second reference moment, the second reference moment is later than a first moment, the first moment is a moment at which the second frame of image is sent based on the second period that uses the first reference moment as a start moment, and the second frame of image is a next frame of image to be received by the terminal.

Optionally, the sending module <NUM> is further configured to send second information to the server when a quantity of frames of an image buffered by the terminal is less than or equal to a second threshold, where the second information indicates to send data of a second frame of image at a second reference moment, the second reference moment is earlier than a first moment, the first moment is a moment at which the second frame of image is sent based on the second period that uses the first reference moment as a start moment, and the second frame of image is a next frame of image to be received by the terminal.

Optionally, the terminal may further include a processing module and a storage module (not shown in <FIG>). The storage module is configured to store data and/or instructions. The processing module may read the data or the instructions in the storage module, to implement the methods corresponding to the foregoing embodiments.

In a possible implementation, the terminal <NUM> is presented in an integrated form of function modules obtained through division. The "module" herein may be an ASIC, a circuit, a processor that executes one or more software or firmware programs, a memory, an integrated logic circuit, and/or another component capable of providing the foregoing functions. In a simple embodiment, a person skilled in the art can figure out that the form of the communication device <NUM> shown in <FIG> may be used for the terminal <NUM>.

Specifically, functions/implementation processes of the sending module <NUM> and the receiving module <NUM> in <FIG> may be implemented by the processor <NUM> in the communication device <NUM> shown in <FIG> by invoking the computer-executable instructions stored in the memory <NUM>. Alternatively, functions/implementation processes of the sending module <NUM> and the receiving module <NUM> in <FIG> may be implemented by the communication interface <NUM> in the communication device <NUM> shown in <FIG>.

Because the terminal <NUM> provided in this embodiment may perform the foregoing video transmission method, for a technical effect that can be achieved by the server, refer to the foregoing method embodiments.

Optionally, an embodiment of this application further provides a communication apparatus (for example, the communication apparatus may be a chip or a chip system). The communication apparatus includes a processor, configured to implement the method in any one of the foregoing method embodiments. In a possible design, the communication apparatus further includes a memory. The memory is configured to store necessary program instructions and necessary data. The processor may invoke program code stored in the memory, to indicate the communication apparatus to perform the method in any one of the foregoing method embodiments. Certainly, the communication apparatus may not include a memory. In another possible design, the communication apparatus further includes an interface circuit. The interface circuit is a code/data read/write interface circuit, and the interface circuit is configured to: receive computer-executable instructions (where the computer-executable instructions are stored in a memory, may be directly read from the memory, or may be read by using another component), and transmit the computer-executable instructions to the processor. When the communication apparatus is a chip system, the communication apparatus may include a chip, or may include a chip and another discrete component. This is not specifically limited in this embodiment of this application.

All or some of the foregoing embodiments may be implemented by using software, hardware, firmware, or any combination thereof. When a software program is used to implement embodiments, embodiments may be implemented completely or partially in a form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the procedure or functions according to embodiments of this application are all or partially generated. The computer may be a general-purpose computer, a dedicated computer, a computer network, or other programmable apparatuses. The computer instructions may be stored in a computer-readable storage medium or may be transmitted from a computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website, computer, server, or data center to another website, computer, server, or data center in a wired (for example, a coaxial cable, an optical fiber, or a digital subscriber line (DSL)) or wireless (for example, infrared, radio, or microwave) manner. The computer-readable storage medium may be any usable medium accessible by a computer, or a data storage device, such as a server or a data center, integrating one or more usable media. The usable medium may be a magnetic medium (for example, a floppy disk, a hard disk, or a magnetic tape), an optical medium (for example, a DVD), a semiconductor medium (for example, a solid state drive (solid state drive, SSD)), or the like. In embodiments of this application, the computer may include the apparatuses described above.

Claim 1:
A video transmission method, comprising:
sending, by a terminal, first field of view information to a server through a first link based on a first period; and
receiving, by the terminal from the server through a second link based on a second period, data of a first frame of image that corresponds to the first field of view information;
wherein the method further comprises:
sending, by the terminal, first information to the server, wherein the first information indicates a first reference moment for sending image data to the terminal, and the first reference moment is a moment at which the server sends the image data to the terminal for the first time; and
a) sending, by the terminal, a second information to the server when a quantity of frames of an image buffered by the terminal is greater than or equal to a first threshold, wherein the second information indicates to send data of a second frame of image at a second reference moment, the second reference moment is later than a first moment, the first moment is a moment at which the second frame of image is sent based on the second period that uses the first reference moment as a start moment, and the second frame of image is a next frame of image to be received by the terminal,
or
b) sending, by the terminal, a second information to the server when a quantity of frames of an image buffered by the terminal is less than or equal to a second threshold, wherein the second information indicates to send data of a second frame of image at a second reference moment, the second reference moment is earlier than a first moment, the first moment is a moment at which the second frame of image is sent based on the second period that uses the first reference moment as a start moment, and the second frame of image is a next frame of image to be received by the terminal.