Patent Description:
More users start to watch videos online through terminals such as a mobile phone, a tablet computer, and a personal computer, and with the development of mobile terminals and network technologies, corresponding content in games can also be displayed to users in a video playing manner. Using cloud gaming as an example, in the cloud gaming scenario, the game does not run on a game terminal of a player, but runs on a cloud server, and the cloud server renders the game scene into a video and audio stream, and transmits the video and audio stream to the terminal of the player through the network. The terminal of the player does not need to have powerful graphics computing and data processing capabilities, but only needs to have basic streaming media playback capabilities and the capability to determine player input commands and transmit the commands to the cloud server.

<CIT> provides a H. <NUM> decoding method, including: receiving first frame of image data after encoding, leading the first frame of image data into a buffer area of a decoder, copying the first frame of image data continuously until the whole buffer area is filled with the first frame of image data; decoding the first frame of image data when the buffer area meets the decoding requirement of the decoder, leading a second frame of image data into the buffer area, decoding the second frame of image data, and repeating the steps until decoding the last frame of image data.

<CIT> describes a video codec to provide repeat padding of hybrid video sequences having arbitrary video resolution. The video codec repeat pads to expand the active content of pictures in the video sequence out to meet an adaptive vertical macroblock alignment restriction that varies by picture type.

To describe the technical solutions of this disclosure more clearly, the following briefly describes the accompanying drawings.

The technical solutions of this disclosure are clearly and completely described below with reference to the accompanying drawings of this disclosure.

In some cloud gaming scenarios, limited by the decoding capability of the terminal of the player, in a decoding solution, a decoding frame cache of video hardware greatly affects the overall delay, which may cause lagging in the finally played video.

This disclosure provides an image processing method and apparatus, an electronic device, and a storage medium, which can reduce time delay caused by decoding an image sequence and improve the smoothness of image processing.

In this disclosure, after padding frame information transmitted by a server and an image frame set of a to-be-processed image sequence are received, the padding frame information being generated by the server according to the image frame set, the image frame set is inputted frame by frame to a cache queue of a decoder and an image frame currently located at the first position of the cache queue is determined as a target image frame. Next, padding frames are inserted between the target image frame and a next image frame subsequent to the target image frame based on the padding frame information, so that the remaining frames other than the target image frame in the cache queue are the padding frames. The next image frame subsequent to the target image frame is used as a new target image frame in response to detecting that the target image frame is decoded, and the step of inserting padding frames between the target image frame and a next image frame subsequent to the target image frame based on the padding frame information is performed, until all image frames are decoded. Finally, the decoded image frames are processed and a processing result is displayed. Therefore, the solution can reduce time delay caused by video decoding, thereby improving the smoothness of image processing.

Specifically, the image processing apparatus may be specifically integrated in a terminal, and the terminal may be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, or the like, but is not limited thereto. The terminal and the server may be directly or indirectly connected in a wired or wireless communication manner. This is not limited in this disclosure.

For example, referring to <FIG>, the image processing apparatus is integrated on a terminal, and the terminal can receive padding frame information transmitted by a server and an image frame set of a to-be-processed image sequence. The padding frame information is generated by the server according to the image frame set, and the server is a cloud server. In the cloud gaming scenario, the cloud server renders the game scene into a video and audio stream (that is, video stream data), and transmits the video and audio stream to a user terminal through the network. Then, the terminal inputs the image frame set frame by frame to a cache queue of a decoder, and determines an image frame currently located at the first position of the cache queue as a target image frame. Next, the terminal inserts padding frames between the target image frame and a next image frame subsequent to the target image frame based on the padding frame information, so that the remaining frames other than the target image frame in the cache queue are the padding frames. Next, the terminal uses the next image frame of the target image frame as a new target image frame in response to detecting that the target image frame is decoded, and performs the operation of inserting padding frames between the target image frame and a next image frame subsequent to the target image frame based on the padding frame information, until all image frames in the image frame set are decoded. Finally, the terminal renders the decoded image frames to play the to-be-processed image sequence.

In the image processing method provided in this disclosure, the padding frames are inserted between the target image frame and the next image frame subsequent to the target image frame, so that the target image frame is located at the first position of the cache queue, and the remaining frames other than the target image frame in the cache queue are the padding frames, which improves the speed of decoding the image frame by the terminal, and reduces time delay caused by video decoding, thereby improving the efficiency of image processing.

Detailed descriptions are separately performed below. A description order of the following examples is not construed as a limitation on a preferred order of the examples.

An image processing method is provided, including: receiving padding frame information transmitted by a server and an image frame set of a to-be-processed image sequence; inputting the image frame set frame by frame to a cache queue of a decoder, and determining an image frame currently located at the first position of the cache queue as a target image frame; inserting padding frames between the target image frame and a next image frame subsequent to the target image frame based on padding frame information; using the next image frame of the target image frame as a new target image frame in response to detecting that the target image frame is decoded, and performing the step of inserting padding frames between the new target image frame and a next image frame subsequent to the new target image frame in the image frame set based on the padding frame information, until all image frames in the image frame set are decoded; and processing the decoded image frames, and displaying a processing result of the decoded image frames.

<FIG> is a schematic flowchart of an image processing method. The image processing method may be performed by a terminal, and a specific process may be as follows:
<NUM>. Receive padding frame information transmitted by a server and an image frame set of a to-be-processed image sequence.

The padding frame information is generated by the server according to the image frame set. The terminal can receive video stream data transmitted by the server through a wired network or wireless network. The video stream data carries the padding frame information and the image frame set of the to-be-processed image sequence, and after the video stream data transmitted by the server is received, the video stream data is parsed to determine the image frame set of the to-be-processed image sequence and the padding frame information generated by the server according to the image frame set. A format of a video data source may be Advanced Video Coding (AVC, that is, H. <NUM> encoding standard) or High Efficiency Video Coding (HEVC, that is, H. <NUM> encoding standard).

First, the following concepts of video encoding are introduced. The so-called video encoding method is compressing an original video image into a binary byte stream through prediction, change, quantization, recombination, and entropy encoding through a compression technology, where encoding methods include: Context-Adaptive Variable-Length Coding (CAVLC) and Context-based Adaptive Binary Arithmetic Coding (CABAC). After determining image sequence header information of a to-be-processed image sequence, the server parses the image sequence header information to determine information such as an image quality level of the to-be-processed image sequence, the quantity of reference frames, whether there are bidirectional prediction frames, and a length corresponding to an image frame sequence, and based on the parsed information, determines whether a padding frame can be generated. In response to generating a padding frame, the server packs the generated padding frame, a sequence number and an offset corresponding to the padding frame into padding frame information, writes the padding frame information into the image sequence header information, and then transmits the image sequence header information including the padding frame information to the terminal.

Input the image frame set frame by frame to a cache queue of a decoder, and determine an image frame currently located at a first position of the cache queue as a target image frame.

To facilitate the storage and transmission of video content, a volume of the video content usually needs to be reduced, that is, an original video image needs to be compressed, and a compression algorithm is also referred to as an encoding format. For example, the server may compress video images by using the H. <NUM> encoding format. In response to the compressed video content transmitted by the server being received, the compressed video content needs to be decompressed, also referred to as decoding. In terms of video image encoding and decoding, an encoder encodes a plurality of pictures to generate a group of pictures (GOP), and during playing, a decoder reads the GOP section by section for decoding, reads the pictures, and then renders the pictures for display. The GOP is a group of continuous pictures, including a key frame and a plurality of non-key frames, where the key frame is a basic frame (the first frame) of the GOP. There is only one key frame in a group, and the non-key frames include a forward reference frame and a bidirectional reference frame. The key frame is a complete picture, and the forward reference frame and the bidirectional reference frame record changes relative to the key frame. Therefore, the key frame may be independently decoded, while the forward reference frame needs to rely on a previous image frame for decoding, and the bidirectional reference frame needs to rely not only on the previous image frame, but also on a next image frame for decoding. That is, in response to a video needing to be played, a position of the key frame needs to be first located, so that the video can be played. That is the step of "inputting the image frame set frame by frame to a cache queue of a decoder, and determining an image frame currently located at the first position of the cache queue as a target image frame" may specifically include:.

Currently, video data (that is, image frame set) can be cached between a user terminal and the server through the cache queue. That is, the cache queue is used to cache each to-be-decoded image frame in a first-in first-out order in an image processing process, and only in response to all cache space in the cache queue being cached with image frames, the terminal decodes the image frame located at the first position of the cache queue.

Insert padding frames between the target image frame and a next image frame subsequent to the target image frame based on the padding frame information.

The padding frames are inserted between the target image frame and the next image frame subsequent to the target image frame, so that the remaining frames other than the target image frame in the cache queue are the padding frames. Using cloud gaming as an example, the cloud gaming, also known as gaming on demand, is an online gaming technology based on a cloud computing technology, and in the cloud gaming, an image frame set transmitted by a server generally includes only a key frame and a forward reference frame. Because a bidirectional reference frame records a difference between a current frame and previous and next frames. That is, to decode the bidirectional reference frame, not only a cached picture before the bidirectional reference frame needs to be determined, but also a picture after the bidirectional reference frame needs to be decoded. A final picture is determined by superimposing the previous and next images with data of the current frame. In response to the bidirectional reference frame needing to be decoded, data of a next frame of the bidirectional reference frame needs to be further determined. In this case, the next frame of the bidirectional reference frame is still in the cache queue, and therefore, the bidirectional reference frame cannot be decoded, causing lagging in the picture.

The padding frame information includes padding frames corresponding to all image frames in the image frame set. Therefore, a padding frame corresponding to the target image frame can be determined according to the padding frame information, and the determined padding frames can be inserted between the target image frame and the next image frame subsequent to the target image frame, so that the target image frame is located at the first position of the cache queue, and the remaining frames other than the target image frame in the cache queue are the padding frames. That is the step of "inserting padding frames between the target image frame and a next image frame subsequent to the target image frame based on the padding frame information" includes:.

The padding frame is a forward prediction frame, and a frame mode of the padding frame is a skip mode. A P-Skip macroblock is a special P macroblock. For general P macroblocks, a pixel residual and a motion vector residual are written into a code stream, and transmitted from an encoding end to a decoding end. But the P-Skip macroblock is special in that neither the pixel residual nor the motion vector residual is transmitted. The encoding end does not need to transmit other information about the macroblock in addition a small quantity of bytes that identify the macroblock as the P-Skip macroblock. Because the motion vector residual is equal to a difference between a motion vector and a predicted motion vector, the motion vector residual at the encoding end is zero, and the predicted motion vector can be determined during decoding. That is, during decoding, the motion vector can also be determined. At the decoding end, there is a reconstructed pixel corresponding to a macroblock of a reference frame. According to the reconstructed pixel and the motion vector, a pixel value of the current macroblock in the current frame can be recovered, which is the so-called P-skip macroblock principle. Literally, the macroblock is skipped, which means that the macroblock is not encoded, and an approximate replacement recovery method is adopted at the decoding end. If a pixel of a macroblock of the current frame and a pixel of a macroblock of the reference frame (the two macroblocks are not required to be in the same position) are almost identical, it is obvious that the macroblock of the current frame does not need to be encoded. At the decoding end, the pixel value of the current macroblock of the current frame can be directly recovered by an approximate substitution method. For example, if there is a ping-pong ball in the first frame, and there is also a ping-pong ball in the second frame, the ping-pong ball macroblock in the second frame is likely to be compiled into a P-Skip macroblock. In this disclosure, the padding frame inserted into the cache queue from the pre-encoding perspective is exactly the same as the target image frame.

Further, to improve the decoding efficiency of the image frame, the padding frames are inserted between the target image frame and the next image frame subsequent to the target image frame, so that the cache queue is padded (up to a maximum cache quantity of the cache queue). A padding quantity can be generated according to the maximum cache quantity of the cache queue and the target image frame. Then, target padding frames with the quantity corresponding to the padding quantity are inserted between the target image frame and the next image frame subsequent to the target image frame. That is, the step of "inserting a preset quantity of target padding frames between the target image frame and the next image frame subsequent to the target image frame" may specifically include:.

To ensure the picture continuity of the to-be-processed image sequence after the padding frames are inserted into the cache queue, frame sequence numbers of the padding frames in the cache queue need to be assigned. That is, the step of "inserting target padding frames with a quantity corresponding to the padding quantity between the target image frame and the next image frame subsequent to the target image frame" specifically further includes:.

For example, the frame sequence number of the target image frame is <NUM>, <NUM> padding frames: a padding frame A, a padding frame B, a padding frame C, and a padding frame D, are inserted into the cache queue, and positions of the padding frames are padding frame A - padding frame B - padding frame C - padding frame D, and corresponding frame sequence numbers of the padding frame A, the padding frame B, the padding frame C, and the padding frame D are assigned in an increasing order. In this case, the frame sequence number of the padding frame A is <NUM>, the frame sequence number of the padding frame B is <NUM>, the frame sequence number of the padding frame C is <NUM>, and the frame sequence number of the padding frame D is <NUM>.

In practical applications, for example, in the H. <NUM> encoding format, an H. <NUM> strip refers to an integer number of macroblocks or macroblock pairs arranged in a raster scan order within a specific strip group. However, these macroblocks or macroblock pairs are not necessarily consecutively arranged in the raster scan order within the image. An address of the macroblock is determined from an address (described in a strip header) of the first macroblock of the strip and a mapping of the macroblock to the strip group. <NUM> strip header is a part of an encoding strip and includes data elements related to the first macroblock or all macroblocks in the strip. That is, an offset corresponding to the padding frames in the cache queue can be extracted from the padding frame information, and the offset refers to an offset of the padding frame in the strip header. Then, based on positions of the frame sequence numbers of the padding frames in the cache queue and the offset corresponding to the padding frames in the cache queue, the frame sequence numbers of the padding frames in the cache queue are assigned. That is, the step of "assigning frame sequence numbers of the padding frames in the cache queue based on the frame sequence number and positions of the padding frames in the cache queue, so that a frame sequence number of the last padding frame in the cache queue is a second frame sequence number" may specifically include:.

Specifically, the frame sequence numbers of the padding frames generated by the server are extracted from the padding frame information, and the frame sequence numbers of the padding frames in the cache queue are updated to corresponding frame sequence numbers according to the frame sequence numbers, the positions of the padding frames in the cache queue, and the offset corresponding to the padding frames in the cache queue, so as to ensure the picture continuity of the to-be-processed image sequence after the padding frames are inserted into the cache queue.

Use the next image frame of the target image frame as a new target image frame in response to detecting that the target image frame is decoded, and perform operation of inserting padding frames between the new target image frame and a next image frame subsequent to the new target image frame in the image frame set based on the padding frame information, until all image frames in the image frame set are decoded.

For example, specifically, in response to detecting that the target image frame is decoded, the next image frame subsequent to the target image frame is used as a new target image frame, that is, the next image frame subsequent to the target image frame is used as a currently processed image frame, and then, padding frames are inserted between the currently processed image frame and a next image frame of the currently processed image frame, that is, the step of inserting padding frames between the target image frame and a next image frame subsequent to the target image frame based on the padding frame information is performed until all the image frames in the image frame set are decoded.

Images represented by different key frames are completely different. Therefore, during insertion of the padding frames, whether the currently processed image frame is a key frame needs to be further detected. That is, the step of "using the next image frame of the target image frame as a new target image frame in response to detecting that the target image frame is decoded, and performing operation of inserting padding frames between the target image frame and a next image frame subsequent to the new target image frame in the image frame set based on the padding frame information, until all image frames in the image frame set are decoded" may specifically include:.

For example, specifically, if the currently processed object is a key frame, the frame sequence number of the currently processed object is determined as a starting sequence number, and frame sequence numbers of the padding frames are assigned in the cache queue based on the frame sequence number and positions of the padding frames in the cache queue; and if the currently processed object is not the key frame, a frame sequence number corresponding to a previous frame of the currently processed object is determined, where the previous frame of the currently processed object is the padding frame, that is, a frame sequence number of the currently processed object is adjusted according to the frame sequence number of the previous padding frame, and frame sequence numbers of the padding frames in the cache queue are assigned based on the adjusted frame sequence number of the currently processed object and positions of the padding frames in the cache queue. If the currently processed object is not the key frame, after the frame sequence numbers of the padding frames in the cache queue are assigned based on the adjusted frame sequence number of the currently processed object and the positions of the padding frames in the cache queue, whether the frame sequence number corresponding to the assigned padding frame is greater than a preset threshold needs to be further detected. If yes, the frame sequence number greater than the preset threshold is reset to zero.

Process the decoded image frames and display a processing result of the decoded image frames.

The decoded image frames can be rendered to play the to-be-processed image sequence. In this disclosure, because the padding frame is a P-skip frame, the P-skip frame is only decoded but not rendered, while each video frame in the image frame set is decoded, rendered, and played, so as to reduce time delay caused by video decoding and improve the smoothness of image processing.

In this disclosure, after padding frame information transmitted by a server and an image frame set of a to-be-processed image sequence are received, the image frame set is inputted frame by frame to a cache queue of a decoder and an image frame currently located at the first position of the cache queue is determined as a target image frame. Next, padding frames are inserted between the target image frame and a next image frame subsequent to the target image frame based on the padding frame information, so that the remaining frames other than the target image frame in the cache queue are the padding frames. The next image frame subsequent to the target image frame is used as a new target image frame in response to detecting that the target image frame is decoded, and the step of inserting padding frames between the target image frame and a next image frame subsequent to the target image frame based on the padding frame information is performed, until all image frames in the image frame set are decoded. Finally, the decoded image frames are processed and a processing result is displayed. In the image processing method provided in this disclosure, the padding frames are inserted between the target image frame and the next image frame subsequent to the target image frame, so that the target image frame is located at the first position of the cache queue, and the remaining frames other than the target image frame in the cache queue are the padding frames, which improves the speed of decoding the image frame by the terminal, and reduces time delay caused by video decoding, thereby improving the efficiency of image processing.

According to the method, the following further provides detailed description by using an example.

In this example the image processing apparatus is specifically integrated in a terminal is used for description.

Referring to <FIG>, a specific process of an image processing method may be as follows:.

In response to receiving compressed video content transmitted by the server, the terminal needs to decompress the compressed video content. That is, in response to needing to play a video, the terminal first needs to locate a position of a key frame, so that the video can be played.

The terminal inserts padding frames between the target image frame and a next image frame subsequent to the target image frame based on the padding frame information.

The padding frame information includes padding frames corresponding to all image frames in the image frame set. Therefore, a padding frame corresponding to the target image frame can be determined according to the padding frame information, and the determined padding frames can be inserted between the target image frame and the next image frame subsequent to the target image frame, so that the target image frame is located at the first position of the cache queue, and the remaining frames other than the target image frame in the cache queue are the padding frames.

The terminal uses the next image frame of the target image frame as a new target image frame in response to detecting that the target image frame is decoded, and performs operation of inserting padding frames between the new target image frame and a next image frame subsequent to the new target image frame in the image frame set based on the padding frame information, until all image frames in the image frame set are decoded.

For example, specifically, if the currently processed object is a key frame, the frame sequence number of the currently processed object is determined as a starting sequence number, and the terminal assigns frame sequence numbers of the padding frames in the cache queue based on the frame sequence number and positions of the padding frames in the cache queue; and if the currently processed object is not the key frame, the terminal determines a frame sequence number corresponding to a previous frame of the currently processed object, where the previous frame of the currently processed object is the padding frame, that is, the terminal adjusts a frame sequence number of the currently processed object according to the frame sequence number of the previous padding frame, and assigns frame sequence numbers of the padding frames in the cache queue based on the adjusted frame sequence number of the currently processed object and positions of the padding frames in the cache queue. If the currently processed object is not the key frame, after assigning the frame sequence numbers of the padding frames in the cache queue based on the adjusted frame sequence number of the currently processed object and the positions of the padding frames in the cache queue, the terminal needs to further detect whether the frame sequence number corresponding to the assigned padding frame is greater than a preset threshold. If yes, the terminal resets the frame sequence number greater than the preset threshold to zero.

The terminal processes the decoded image frames and displays a processing result of the decoded image frames.

For example, specifically, the decoded image frames are rendered to play the to-be-processed image sequence. In this disclosure, because the padding frame is a P-skip frame, the P-skip frame is only decoded but not rendered, while each video frame in the image frame set is decoded, rendered, and played, so as to reduce time delay caused by video decoding and improve the smoothness of image processing.

In this disclosure, after receiving padding frame information transmitted by a server and an image frame set of a to-be-processed image sequence, the terminal inputs the image frame set frame by frame to a cache queue of a decoder and determines an image frame currently located at the first position of the cache queue as a target image frame. Next, the terminal inserts padding frames between the target image frame and a next image frame subsequent to the target image frame based on the padding frame information, so that the remaining frames other than the target image frame in the cache queue are the padding frames. The terminal uses the next image frame of the target image frame as a new target image frame in response to detecting that the target image frame is decoded, and performs the step of inserting padding frames between the new target image frame and a next image frame subsequent to the new target image frame in the image frame set based on the padding frame information, until all image frames in the image frame set are decoded. Finally, the terminal processes the decoded image frames and displays a processing result of the decoded image frames. The terminal provided in this disclosure inserts the padding frames between the target image frame and the next image frame subsequent to the target image frame, so that the target image frame is located at the first position of the cache queue, and the remaining frames other than the target image frame in the cache queue are the padding frames, which improves the speed of decoding the image frame by the terminal, and reduces time delay caused by video decoding, thereby improving the efficiency of image processing.

To further understand the image processing solution of this disclosure, referring to <FIG>, this disclosure provides an image processing system (hereinafter referred to as the processing system). The processing system includes: a first terminal <NUM>, a second terminal <NUM>, and a server <NUM>. After collecting and encoding a to-be-processed image sequence to generate an H. <NUM> video stream, the terminal <NUM> transmits image sequence header information of a to-be-played video to the server <NUM>. The image sequence header information includes a sequence parameter set and an image parameter set of the to-be-played video. The sequence parameter set stores a set of global parameters of an encoded video sequence. The so-called encoded video sequence is a sequence including encoded pixel data of frames of an original video. Parameters on which encoded data of each frame depends are stored in the image parameter set. Then, after receiving the image sequence header information, the server <NUM> parses the image sequence header information to determine basic format information of the to-be-processed image sequence, including the quantity of reference frames, a video level, a maximum frame sequence, an offset of a frame sequence number in a corresponding strip, and the quantity of image frames of bidirectional reference frames, and determines, based on a parsing result, whether the to-be-processed image sequence can generate a padding frame. Specifically, referring to <FIG>, in response to no bidirectional reference frame in the to-be-processed image sequence and the maximum quantity of reference frames being <NUM>, it is determined that the to-be-processed image sequence can generate a padding frame; otherwise, the process ends. Further, in response to determining that the to-be-processed image sequence can generate the padding frame, a corresponding padding frame generation module is determined according to the parsing result. For example, if an entropy encoding mode is a CAVLC mode, a padding frame generation module corresponding to the CAVLC mode is selected. If the entropy encoding mode is a CABAC mode, a padding frame generation module corresponding to the CABAC mode is selected. Then based on the selected padding frame generation module, an encoder is specified to generate a forward reference frame without any motion vector estimation. All macroblock prediction modes are forced to be a P-skip mode (skip mode), and corresponding padding frames are generated by specifying the entropy encoding mode (CAVLC/CABAC). Finally, a padding frame information packaging module encapsulates the generated padding frame, the frame sequence number in the parsing result, and the offset of the frame sequence number in the corresponding strip to determine the padding frame information. The server <NUM> transmits the image sequence header information and the padding frame information to the second terminal <NUM>.

Referring to <FIG>, the second terminal <NUM> receives the image sequence header information and the padding frame information transmitted by the server <NUM>, and the second terminal <NUM> parses the image sequence header information to determine the image frame set of the to-be-processed image sequence. Then the second terminal <NUM> inserts the image frame set frame by frame to a decoding module of the second terminal <NUM>. If there is a cache queue during decoding, the second terminal <NUM> extracts the padding frame generated by the server <NUM> from the padding frame information. Next, the second terminal <NUM> determines, according to the image frame set, that an image frame currently at the first position of the cache queue is a target image frame, and inserts the padding frames between the target image frame and a next image frame subsequent to the target image frame based on the padding frame information. Then, the second terminal <NUM> respectively updates frame sequence numbers of the padding frame and the target image frame in the cache queue according to the offset in the padding frame information, so as to ensure the picture continuity of the to-be-processed image sequence. For details, refer to the foregoing examples, which are not repeated herein. In addition, if there is no cache queue during decoding, a padding frame generation module in the server <NUM> transmits the image sequence header information to a relay proxy server, so that the second terminal <NUM> receives the image sequence header information transmitted by the relay proxy server and displays the to-be-processed image sequence. If the decoded video frame is a padding frame, the padding frame is not rendered, and a decoding step is performed to play the to-be-played video.

To help better implement the image processing method of this disclosure, this disclosure further provides an image processing apparatus based on the foregoing method. Terms have meanings the same as those in the foregoing image processing method. For specific implementation details, reference may be made to the description in the method examples.

<FIG> is a schematic structural diagram of an image processing apparatus according to this disclosure. The image processing apparatus may include a receiving module <NUM>, a determining module <NUM>, an insertion module <NUM>, a detection module <NUM>, and a rendering module <NUM>, which may be specifically as follows:
The receiving module <NUM> is configured to receive padding frame information transmitted by a server and an image frame set of a to-be-processed image sequence.

The padding frame information is generated by the server according to the image frame set. The receiving module <NUM> can receive video stream data transmitted by the server through a wired network or wireless network. The video stream data carries the padding frame information and the image frame set of the to-be-processed image sequence, and after the video stream data transmitted by the server is received, the video stream data is parsed to determine the image frame set of the to-be-processed image sequence and the padding frame information generated by the server according to the image frame set. A format of a video data source may be AVC (that is, H. <NUM> encoding standard) or HEVC (that is, H. <NUM> encoding standard).

The determining module <NUM> is configured to input the image frame set frame by frame to a cache queue of a decoder, and determine an image frame currently located at a first position of the cache queue as a target image frame.

To facilitate the storage and transmission of video content, a volume of the video content usually needs to be reduced, that is, an original video image needs to be compressed, and a compression algorithm is also referred to as an encoding format. For example, a video image can be compressed by using the H. <NUM> encoding format. In response to the determining module <NUM> receiving compressed video content transmitted by the server, the terminal needs to decompress the compressed video content. That is, in response to needing to play a video, the determining module <NUM> first needs to locate a position of a key frame, so that the video can be played.

The determining module <NUM> may be specifically configured to: determine a key frame of a currently rendered to-be-processed image sequence to determine a target key frame, determine a next image frame of the target key frame in the image frame set as a target image frame, and store the target image frame at the first position of the cache queue.

The insertion module <NUM> is configured to insert padding frames between the target image frame and a next image frame subsequent to the target image frame based on the padding frame information.

The padding frame information includes padding frames corresponding to all image frames in the image frame set. Therefore, the insertion module <NUM> can determine a padding frame corresponding to the target image frame according to the padding frame information, and insert the determined padding frames between the target image frame and the next image frame subsequent to the target image frame, so that the target image frame is located at the first position of the cache queue, and the remaining frames other than the target image frame in the cache queue are the padding frames.

The insertion module <NUM> may specifically include:.

The insertion unit may be specifically configured to: detect a maximum cache quantity corresponding to the cache queue, generate a padding quantity according to the target image frame and the maximum cache quantity, and insert target padding frames with a quantity corresponding to the padding quantity between the target image frame and the next image frame subsequent to the target image frame.

Referring to <FIG>, the image processing apparatus may further include an assignment module <NUM>, and the assignment module <NUM> includes:.

The assignment unit may be specifically configured to: extract an offset corresponding to the padding frames in the cache queue from the padding frame information, and assign the frame sequence numbers of the padding frames in the cache queue based on the frame sequence number, the positions of the padding frames in the cache queue, and the offset corresponding to the padding frames in the cache queue.

The detection module <NUM> is configured to: use the next image frame of the target image frame as a new target image frame in response to detecting that the target image frame is decoded, and perform operation of inserting padding frames between the new target image frame and a next image frame subsequent to the new target image frame in the image frame set based on the padding frame information, until all image frames in the image frame set are decoded.

For example, specifically, if a currently processed object is a key frame, a frame sequence number of the currently processed object is determined as a starting sequence number, and the detection module <NUM> assigns frame sequence numbers of the padding frames in the cache queue based on the frame sequence number and positions of the padding frames in the cache queue; and if the currently processed object is not the key frame, the terminal determines a frame sequence number corresponding to a previous frame of the currently processed object, where the previous frame of the currently processed object is the padding frame, that is, the detection module <NUM> adjusts a frame sequence number of the currently processed object according to the frame sequence number of the previous padding frame, and assigns frame sequence numbers of the padding frames in the cache queue based on the adjusted frame sequence number of the currently processed object and positions of the padding frames in the cache queue. If the currently processed object is not the key frame, after assigning the frame sequence numbers of the padding frames in the cache queue based on the adjusted frame sequence number of the currently processed object and the positions of the padding frames in the cache queue, the detection module <NUM> further needs to detect whether the frame sequence number corresponding to the assigned padding frame is greater than a preset threshold. If yes, the terminal resets the frame sequence number greater than the preset threshold to zero.

The detection module <NUM> may be specifically configured to: use the next image frame subsequent to the target image frame as the new target image frame in response to detecting that the target image frame is detected, to determine a currently processed object, and detect a type of the currently processed object, insert the padding frames between the currently processed object and a next image frame of the currently processed object based on the padding frame information by using a frame sequence number of the currently processed object as a benchmark, in response to a detection result indicating that the currently processed object is a key frame; adjust the frame sequence number of the currently processed object according to a frame sequence number corresponding to a previous frame of the currently processed object in response to detecting that the target image frame is not the key frame, and insert the padding frames between the currently processed object and the next image frame of the currently processed object based on the padding frame information; and perform the operation of using the next image frame of the target image frame as the new target image frame to determine a currently processed object, until all the image frames in the image frame set are decoded.

The rendering module <NUM> is configured to render the decoded image frames to play the to-be-processed image sequence.

In this disclosure, because the padding frame is a P-skip frame, the rendering module <NUM> only decodes but not renders the P-skip frame, while each video frame in the image frame set is decoded, rendered, and played, so as to reduce time delay caused by video decoding and improve the smoothness of image processing.

In this disclosure, after the receiving module <NUM> receives padding frame information transmitted by a server and an image frame set of a to-be-processed image sequence, the determining module <NUM> inputs the image frame set frame by frame to a cache queue of a decoder and determines an image frame currently located at the first position of the cache queue as a target image frame. Next, the insertion module <NUM> inserts padding frames between the target image frame and a next image frame subsequent to the target image frame based on the padding frame information, so that remaining frames other than the target image frame in the cache queue are the padding frames. The detection module <NUM> uses the next image frame of the target image frame as a new target image frame in response to detecting that the target image frame is decoded, and performs the step of inserting padding frames between the new target image frame and a next image frame subsequent to the new target image frame in the image frame set based on the padding frame information, until all image frames in the image frame set are decoded. Finally, the rendering module <NUM> renders the decoded image frames to play the to-be-processed image sequence. In the image processing apparatus provided in this disclosure, the padding frames are inserted between the target image frame and the next image frame subsequent to the target image frame, so that the target image frame is located at the first position of the cache queue, and the remaining frames other than the target image frame in the cache queue are the padding frames, which improves the speed of decoding the image frame by the terminal, and reduces time delay caused by video decoding, thereby improving the efficiency of image processing.

In addition, this disclosure further provides an electronic device. <FIG> is a schematic structural diagram of the electronic device according to this disclosure. Specifically,
the electronic device may include components such as a processor <NUM> with one or more processing cores, a memory <NUM> with one or more computer-readable storage medium, a power supply <NUM>, and an input unit <NUM>. A person skilled in the art may understand that the electronic device structure shown in <FIG> does not constitute a limitation to the electronic device.

The processor <NUM> is a control center of the electronic device, and connects various parts of the entire electronic device by using various interfaces and lines. By running or executing a software program and/or a module stored in the memory <NUM>, and invoking data stored in the memory <NUM>, the processor performs various functions of the electronic device and processes data, thereby performing overall monitoring on the electronic device. The processor <NUM> may include one or more processing cores. The processor <NUM> may integrate an application processor and a modem processor. The application processor mainly processes an operating system, a user interface, an application program, and the like. The modem processor mainly processes wireless communication. It can be understood that the foregoing modem processor may alternatively not be integrated into the processor <NUM>.

The memory <NUM> may be configured to store a software program and a module, and the processor <NUM> runs the software program and the module that are stored in the memory <NUM>, to implement various functional applications and data processing. The memory <NUM> may mainly include a program storage area and a data storage area. The program storage area may store an operating system, an application program required by at least one function (for example, a sound playback function and an image playback function), or the like; and the data storage area may store data created according to use of the electronic device. In addition, the memory <NUM> may include a high speed random access memory, and may further include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory, or another transitory solid-state storage device. Correspondingly, the memory <NUM> may further include a memory controller, to provide access of the processor <NUM> to the memory <NUM>.

The electronic device further includes the power supply <NUM> for supplying power to the components. The power supply <NUM> may logically connect to the processor <NUM> by using a power supply management system, thereby implementing functions, such as charging, discharging, and power consumption management, by using the power supply management system. The power supply <NUM> may further include one or more direct current or alternating current power supplies, a re-charging system, a power failure detection circuit, a power supply converter or inverter, a power supply state indicator, and any other component.

The electronic device may further include the input unit <NUM>. The input unit <NUM> may be configured to receive inputted numeric or character information and generate keyboard, mouse, joystick, optical, or trackball signal input related to user settings and function control.

Although not shown in the figure, the electronic device may further include a display unit, and the like. Specifically, the processor <NUM> of the electronic device may load, according to the following instructions, executable files corresponding to processes of one or more application programs into the memory <NUM>. The processor <NUM> runs the application programs stored in the memory <NUM>, to implement the various functions as follows:
receiving padding frame information transmitted by a server and an image frame set of a to-be-processed image sequence; inputting the image frame set frame by frame to a cache queue of a decoder, and determining an image frame currently located at the first position of the cache queue as a target image frame; inserting padding frames between the target image frame and a next image frame subsequent to the target image frame based on padding frame information; using the next image frame of the target image frame as a new target image frame in response to detecting that the target image frame is decoded, and performing the step of inserting padding frames between the new target image frame and a next image frame subsequent to the new target image frame in the image frame set based on the padding frame information, until all image frames are decoded; and rendering the decoded image frames to play the to-be-processed image sequence.

For specific implementation of the foregoing operations, reference may be made to the foregoing examples.

In this disclosure, after padding frame information transmitted by a server and an image frame set of a to-be-processed image sequence are received, the image frame set is inputted frame by frame to a cache queue of a decoder and an image frame currently located at the first position of the cache queue is determined as a target image frame. Next, padding frames are inserted between the target image frame and a next image frame subsequent to the target image frame based on the padding frame information, so that the remaining frames other than the target image frame in the cache queue are the padding frames. The next image frame subsequent to the target image frame is used as a new target image frame in response to detecting that the target image frame is decoded, and the step of inserting padding frames between the target image frame and a next image frame subsequent to the target image frame based on the padding frame information is performed until all image frames are decoded. Finally, the decoded image frames are rendered to play the to-be-processed image sequence. In the image processing method provided in this disclosure, the padding frames are inserted between the target image frame and the next image frame subsequent to the target image frame, so that the target image frame is located at the first position of the cache queue, and the remaining frames other than the target image frame in the cache queue are the padding frames, which improves the speed of decoding the image frame by the terminal, and reduces time delay caused by video decoding, thereby improving the efficiency of image processing.

A person of ordinary skill in the art may understand that, all or some steps of the methods in the foregoing examples may be implemented by using instructions, or implemented through instructions controlling relevant hardware, and the instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

Accordingly, this disclosure provides a non-transitory computer-readable storage medium, storing a plurality of instructions. The instructions can be loaded by the processor, to perform the steps in any image processing method according to this disclosure. For example, the instructions may perform the following steps:
receiving padding frame information transmitted by a server and an image frame set of a to-be-processed image sequence; inputting the image frame set frame by frame to a cache queue of a decoder, and determining an image frame currently located at the first position of the cache queue as a target image frame; inserting padding frames between the target image frame and a next image frame subsequent to the target image frame based on padding frame information; using the next image frame of the target image frame as a new target image frame in response to detecting that the target image frame is decoded, and performing the step of inserting padding frames between the new target image frame and a next image frame subsequent to the new target image frame in the image frame set based on the padding frame information, until all image frames are decoded; and rendering the decoded image frames to play the to-be-processed image sequence.

The storage medium may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk, an optical disc, or the like.

Because the instructions stored in the storage medium may perform the steps of any image processing method provided in this disclosure, the instructions can implement beneficial effects that may be implemented by any image processing method in this disclosure. For details, refer to the foregoing examples.

The system related to this disclosure may be a distributed system formed by connecting a client to a plurality of nodes (computing devices in any form in an access network, such as, servers and user terminals) in a network communication form.

For example, the distributed system is a blockchain system. Referring to <FIG> is a schematic structural diagram of a distributed system <NUM> applied to a blockchain system according to this disclosure. The distributed system is formed of a plurality of nodes (computing devices in any form in an access network, such as, servers and user terminals) and a client. A peer-to-peer (P2P) network is formed between the nodes. The P2P protocol is an application-layer protocol running over the Transmission Control Protocol (TCP). Any machine such as a server or a terminal may be added to the distributed system to become a node. The nodes include a hardware layer, an intermediate layer, an operating system layer, and an application layer.

Referring to functions of each node in the blockchain system shown in <FIG>, the related functions include the following:.

In addition to the routing function, the node may further have the following functions:
(<NUM>) Application: which is deployed in a blockchain, and is used for implementing a particular service according to an actual service requirement, recording data related to function implementation to form recorded data, adding a digital signature to the recorded data to indicate a source of task data, and transmitting the recorded data to another node in the blockchain system, so that the another node adds the recorded data to a temporary block in response to successfully verifying a source and integrity of the recorded data.

For example, a service implemented by this disclosure may include: blockchain: including a series of blocks that are consecutive in a chronological order of generation. Once a new block is added to the blockchain, the new block is no longer removed. The block records recorded data submitted by the node in the blockchain system.

Claim 1:
An image processing method, comprising:
receiving (<NUM>, <NUM>) padding frame information transmitted by a server and an image frame set of a to-be-processed image sequence, the padding frame information being generated by the server according to the image frame set;
inputting (<NUM>, <NUM>) the image frame set frame by frame to a cache queue of a decoder, and determining (<NUM>, <NUM>) an image frame currently located at a first position of the cache queue as a target image frame;
inserting (<NUM>, <NUM>) padding frames between the target image frame and a next image frame subsequent to the target image frame based on the padding frame information, so that the target image frame is located at the first position of the cache queue, and remaining frames other than the target image frame in the cache queue are the padding frames;
using (<NUM>, <NUM>) the next image frame of the target image frame as a new target image frame in response to detecting that the target image frame and remaining padding frames are decoded, and performing (<NUM>, <NUM>) operation of inserting padding frames between the new target image frame and a next image frame subsequent to the new target image frame in the image frame set based on the padding frame information, until all image frames in the image frame set are decoded; and
rendering the decoded image frames and displaying (<NUM>, <NUM>) a rendering result of the decoded image frames, wherein the padding frames are only decoded but not rendered;
wherein the inserting padding frames between the target image frame and a next image frame subsequent to the target image frame based on the padding frame information comprises:
extracting a padding frame set generated by the server according to the image frame set from the padding frame information;
selecting a padding frame corresponding to the target image frame in the padding frame set to determine a target padding frame; and
inserting a preset quantity of target padding frames between the target image frame and the next image frame subsequent to the target image frame.