Patent Description:
An aspect ratio, also known as a height-width ratio, of a source video and an aspect ratio of a display of a current video playing device may be inconsistent. For example, videos shot by users, downloaded from the Internet, etc. may have multiple aspect ratios, and the displays of the users' playing devices may also have multiple aspect ratios. If the aspect ratio of a source video is inconsistent with the aspect ratio of a target display, when the video is played on the target display (for example, a video of which the aspect ratio is <NUM>:<NUM> is played on the target display of which the aspect ratio is <NUM>:<NUM>), large black screens may appear on both sides of the display.

In the related art, in order to match the aspect ratio of the played source video with the aspect ratio of the display of the playing device, so as to fill the display with a video picture to provide better user experience, it is usually necessary to scale or clip the video picture.

Related technology is known from <CIT>, disclosing automated video cropping by minimizing an objective function, the objective function accounting for loss due to cropping, scaling and jitter between adjacent frames.

In order to overcome the problems in a related art, the disclosure provides a video processing method, a video processing device, and a storage medium.

The features of the video processing method and device, and the storage medium according to the present disclosure are defined in the independent claims, and the preferable features according to the present invention are defined in the dependent claims.

The technical solutions in the embodiments of the disclosure may have the following beneficial effects: each of the plurality of video frames extracted from the source video is respectively clipped multiple times to generate the plurality of candidate clipping boxes with the second aspect ratio, the clipping box is selected based on the scores of the playing effects of the candidate clipping boxes, and finally the video frames are clipped by using the clipping box to generate the target video with the second aspect ratio. Therefore, the source video may be output according to a target aspect ratio, and the playing effect of the output content may be ensured.

It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, but do not limit the disclosure.

The implementations set forth in the following description of exemplary embodiments do not represent all implementations consistent with the disclosure. Instead, they are merely examples of devices and methods consistent with aspects related to the disclosure as recited in the appended claims.

In the related art, in order to match an aspect ratio of a played source video with an aspect ratio of a display of a playing device so as to fill the display with a video picture to provide better user experience, it is usually necessary to scale or clip the video picture. In a solution of scaling the video picture, the source video is directly scaled to have a target aspect ratio. Since the aspect ratio of the source video is inconsistent with the target aspect ratio, horizontal and vertical scaling factors of the video are necessarily different. This will result in a certain degree of stretching or compression of the picture, which is more obvious when the difference between the two aspect ratios is larger, such that the impression is affected. In another solution, content-sensitive scaling is used. Firstly, important objects/contents in the video are detected and picked out; and then the objects/contents are pasted onto the background which is scaled to have the target aspect ratio. However, the overall processes involve multiple operations such as important object detection, image restoration, scaling, image synthesis, and the like and, therefore the complexity of calculation may be higher. In a solution of clipping the video picture, a common manner is to adopt a center clipping method of retaining a center of the video picture, and output the source video to a target display in real time so as to achieve full-screen playing. In the overall implementation processes, there is no unequal scaling of the contents in vertical and horizontal directions, and all the displayed contents including the background will not be stretched or compressed. However, it is difficult to ensure that key contents of the video are not lost by cutting off redundant contents at both ends of the video and only retaining the center of the video, and it is impossible to maintain the beauty of the clipped picture.

In view of the above, embodiments of the disclosure provide a video processing method. In combination with clipping of fixed areas and video processing of content-sensitive areas, a source video is output according to a target aspect ratio, and the playing effect of output contents may be ensured.

In the embodiments of the disclosure, a plurality of video frames are extracted from a source video, each of the plurality of extracted video frames is respectively clipped multiple times to generate a plurality of candidate clipping boxes with a target aspect ratio, a clipping box is selected based on scores of playing effects of the candidate clipping boxes, and finally the video frames are clipped by using the clipping box to generate a target video with the target aspect ratio. Therefore, the source video may be output according to the target aspect ratio, and the playing effect of the output content may be ensured.

<FIG> is a flow chart showing a video processing method according to an embodiment. As illustrated in <FIG>, the video processing method includes operations as follows.

At S11, a plurality of video frames are extracted from a source video with a first aspect ratio.

In the embodiment of the disclosure, the source video may be understood as a video to be processed, and has a certain aspect ratio. For convenience of description, the aspect ratio of the source video is referred to herein as the first aspect ratio.

In the embodiment of the disclosure, the plurality of video frames are extracted from the source video with the first aspect ratio, so as to be converted into images for subsequent clipping. When the plurality of video frames are extracted from the source video, all video frames in the source video may be extracted, or the video frames may be sparsely extracted at equal intervals.

In the embodiment of the disclosure, the number of the plurality of extracted video frames is marked as N, and N is a positive integer.

At S12, each of the plurality of video frames is respectively clipped multiple times to generate a plurality of candidate clipping boxes with a second aspect ratio.

In the embodiment of the disclosure, a rectangular box with a target aspect ratio may be generated at a fixed step size on a full video frame in manner of a sliding window as a candidate clipping box. For convenience of description, the target aspect ratio is referred to herein as the second aspect ratio.

In the embodiment of the disclosure, the number of the plurality of candidate clipping boxes generated by clipping each of the video frames multiple times is marked as M, and M is a positive integer.

In the embodiment of the disclosure, the number of the candidate clipping boxes may be set according to actual needs. In order to reduce the number of the candidate clipping boxes and avoid a large number of contents from being clipped out, the size of the sliding window may be limited to be greater than or equal to a preset threshold.

At S13, scores of the plurality of candidate clipping boxes of each of the plurality of video frames are respectively determined, and the scores represent playing effects of the video frames when each of the candidate clipping boxes is taken as a clipping range.

In order to determine content-sensitive areas, the playing effect when each of the candidate clipping boxes is taken as the clipping range may be scored. High or low scores represent the playing effects of the video frames when each of the candidate clipping boxes is taken as the clipping range.

The scores of all candidate clipping boxes of each of the video frames may be adjusted, such that there is a clipping box in each of the video frames. A softmax function may be used to adjust the sum of the scores of all candidate clipping boxes of each of the video frames to be <NUM>, such that the weighting score of each of the candidate clipping boxes may represent the playing effect of the video frame when each of the candidate clipping boxes is taken as the clipping range.

In the embodiment of the disclosure, the score of the mth candidate clipping box of the nth frame may be marked as Sn,m, where n is a positive integer greater than or equal to <NUM> and less than or equal to N, and m is a positive integer greater than or equal to <NUM> and less than or equal to M.

At S14, a clipping box of each of the video frames is selected from the plurality of candidate clipping boxes based on the scores.

At S15, each of the video frames is clipped by using the selected clipping box, and a target video with the second aspect ratio is generated.

In the embodiment of the disclosure, each of the plurality of video frames extracted from the source video with the first aspect ratio is respectively clipped multiple times to generate the plurality of candidate clipping boxes with the second aspect ratio, the clipping box is selected based on the scores of the playing effects of the candidate clipping boxes, and the video frames are clipped by using the selected clipping box to generate the target video with the second aspect ratio. Therefore, the source video may be output according to the target aspect ratio, and the playing effect of the output content may be ensured.

In the following embodiments of the disclosure, the video processing procedures in the above embodiments will be described in conjunction with practical applications.

In the embodiments of the disclosure, firstly, the process of scoring the candidate clipping boxes is described.

In the embodiments of the disclosure, for the N video frames extracted from the source video, the scores of the candidate clipping boxes may be determined frame by frame.

In an embodiment, when the scores of the candidate clipping boxes are determined, image contents in the candidate clipping boxes may be aesthetically scored to obtain the scores of the candidate clipping boxes. In the embodiments, specific implementation processes of aesthetic scoring are not limited. An existing image aesthetic clipping method or image aesthetic scoring method may be used to aesthetically score all M candidate clipping boxes of each of the frames. Then, a softmax function is used to adjust the sum of the scores of all candidate clipping boxes of each of the video frames to be <NUM>, such that scores of all M candidate clipping boxes for each of the frames may be obtained. For example, the score Sn,m of the mth candidate clipping box of the nth frame may be obtained.

In another embodiment, when the scores of the candidate clipping boxes are determined, the scoring process may also be performed based on a salient target detection manner. In the embodiment, for each of the candidate clipping boxes in all M candidate clipping boxes of each of the frames, the scores are determined respectively in the following manner: salient target detection is performed for each of pixels in the candidate clipping boxes, and a pixel score of each of the pixels is determined based on a salient target detection result. The scores of the candidate clipping boxes are determined according to the pixel score of each of the pixels in the candidate clipping boxes.

In the disclosure, when a video is played based on the clipping box selected from the candidate clipping boxes of each of the video frames, the problem of frame-to-frame jitter may occur. When the clipping box of each of the video frames is selected from the plurality of candidate clipping boxes based on the scores, the jitter between adjacent frames may be smoothly controlled.

In the disclosure, the process of selecting the clipping box of each of the video frames from the plurality of candidate clipping boxes may be regarded as a classification process, and a linear chain conditional random fields (linear-CRF) model is adopted for processing to eliminate the jitter between adjacent frames.

<FIG> is a flow chart for implementing smooth control between frames by selecting a clipping box of each of video frames from a plurality of candidate clipping boxes based on scores according to an embodiment. As illustrated in <FIG>, the smooth control on the jitter between adjacent frames includes operations as follows.

At S141, a state transition matrix is constructed based on a position between every two candidate clipping boxes for each of the plurality of video frames.

The state transition matrix constructed according to the embodiments of the disclosure is configured to smoothly control a jitter between adjacent frames for the selected candidate clipping box of each of the video frames.

In the embodiments of the disclosure, the state transition matrix is set according to the difference in a size of a space between the candidate clipping boxes. The state transition matrix may be expressed as: W ∈ RM×M.

In an embodiment, the state transition matrix is constructed based on an IoU between every two candidate clipping boxes and a coefficient multiplier.

If the calculation is based on the IoU between two clipping boxes, a calculation formula may be as follows:
<MAT>
where Wi, j represents a value of the ith row and jth column in the state transition matrix W; Ri represents the ith candidate clipping box; IoU( Ri, Rj) represents calculation of an IoU between the ith candidate clipping box and the jth candidate clipping box; and α is a coefficient multiplier and is configured to control a degree of smoothing, and the degree of smoothing may be controlled by adjusting the size of α.

In another embodiment, the state transition matrix is constructed based on a Euclidean distance between center positions of every two adjacent candidate clipping boxes, areas of the candidate clipping boxes and a coefficient multiplier in the embodiments of the disclosure; and the coefficient multiplier is configured to control a degree of smoothing.

For example, the sizes and center positions of the clipping boxes are considered at the same time, and the following formula may be used to determine the state transition matrix:
<MAT>
where Ceni represents center coordinates of the ith candidate clipping box; Areai represents an area of the ith candidate clipping box; a Diff(Ceni, Cenj) function is configured to calculate the difference between the center coordinates of two clipping boxes, where the difference may be calculated directly by using a Euclidean distance; λ is configured to balance influences of positions of center points and difference of areas, and the influences of change of the areas and change of the positions of the center points may be balanced by adjusting λ; and α is a coefficient multiplier and is configured to control the degree of smoothing, and the degree of smoothing may be controlled by adjusting the size of α.

At S142, a clipping box sequence which meets a maximum clipping box score is determined by using the state transition matrix and the score of each of the candidate clipping boxes based on a maximum target function, and the clipping box sequence includes the selected candidate clipping box of each of the video frames.

In an embodiment, a Viterbi algorithm is used to calculate the following maximum target function, so as to obtain a clipping box sequence Y = {y<NUM>,y<NUM>,. , yN} without frame-to-frame jitter:
<MAT>
where yi ∈ [<NUM>, M] represents a serial number of the selected candidate clipping box of the ith frame; Sn,yn represents a score of the selected yth candidate clipping box of the nth frame; and Wyn-<NUM>,yn represents a state transition matrix value corresponding to the selected yth candidate clipping box of the nth frame and the selected yth candidate clipping box of the (n-<NUM>)th frame.

In the embodiments of the disclosure, after the above anti-jitter smoothing processing, each of the clipping boxes may have no jitter between frames. Generally, the position and size of a candidate clipping box remain unchanged within a period of time, but the candidate clipping box may suddenly jump to another candidate clipping box and remain for a period of time. Such jump of the clipping box may cause the jump and incoherence of the finally output video content. Therefore, smoothing for the change of such clipping box may be performed, including smoothing of the movement and scaling of the clipping box, so as to adjust the position of the clipping box.

<FIG> is a flow chart showing a method for adjusting positions of clipping boxes according to an exemplary embodiment. As illustrated in <FIG>, the method for adjusting positions of clipping boxes includes operations as follows.

At S21, the source video is divided into a plurality of video segments with an unchanged position and size of the selected clipping box between adjacent frames.

In the embodiments of the disclosure, each of the video segments with the unchanged position and size of the clipping box may be determined. It is assumed that a total number of the plurality of video segments is K, and K is a positive integer greater than <NUM>.

At S22, a speed for the clipping boxes moving in two adjacent video segments is determined based on a middle time point of each of the two adjacent video segments and a distance between the clipping boxes in the two adjacent video segments.

In the embodiment, it is assumed that the duration of the kth video is Tk, and the middle time point is tk, where k ∈ [<NUM>,K]. A vector Boxk = [x<NUM>k,y<NUM>k,x<NUM>k,y<NUM>k] is configured to represent the clipping box of the k ∈ [<NUM>, k] th video, where x<NUM>k,y<NUM>k are respectively horizontal and vertical coordinates of an upper left vertex of the clipping box, and x<NUM>k,y<NUM>k are respectively horizontal and vertical coordinates of a lower right vertex of the clipping box.

When the movement speed of the clipping box is determined, the following formula may be used to calculate the speed at which the clipping box moves from the kth segment to the (k+<NUM>)th segment:
<MAT>
where the subtraction of the clipping box refers to the subtraction of four coordinates respectively. Each value in the calculated speed vk = [v<NUM>k,v<NUM>k,v<NUM>k,v<NUM>k] corresponds to the movement speed of each of the coordinates along a specific axis, that is, v<NUM>k,v<NUM>k are respectively movement speeds of the upper left vertex and the lower right vertex along an x axis, and v<NUM>k,v<NUM>k are respectively movement speeds of the upper left vertex and the lower right vertex along a y axis.

At S23, positions of the clipping boxes in the two adjacent video segments are adjusted based on the speed for the clipping box moving in the two adjacent video segments and playing time.

In the embodiment, for a first video segment, if the playing time is less than a middle time point of the first video segment, a position of the clipping box in the first video segment is kept unchanged.

When k ∈ [<NUM>, K - <NUM>], for a kth video segment and a (k+<NUM>)th video segment, if the playing time is greater than or equal to a middle time point of the kth video segment and less than a middle time point of the (k+<NUM>)th video segment, the clipping box in the kth video segment is adjusted to move a first distance; the first distance is a product of a first speed and a first time, the first speed is a speed at which the clipping box moves from the kth video segment to the (k+<NUM>)th video segment, and the first time is a difference between a current time and the middle time point of the kth video segment.

For a Kth video segment, if the playing time is greater than a middle time point of the Kth video segment, a position of the clipping box in the Kth video segment is kept unchanged.

In the embodiments of the disclosure, for any time t, the following formula may be used to calculate the position Bt of the clipping box at this time:
<MAT>
where if t < t<NUM> (that is, the front half segment of the first segment of video),<IMG> Bt = Box<NUM>. If t > tK (that is, the rear half segment of the last segment of video), Bt = BoxK. For k ∈ [<NUM>,K - <NUM>], tk ≤ t < tk+<NUM>, Bt = Boxk + vk × (t - tk).

In the embodiments of the disclosure, each of the video frames in the source video may be clipped in the above manner to generate clipping boxes, and the clipping boxes generated by clipping have the same aspect ratio. The clipping boxes having the same aspect ratio are scaled to have the same resolution and then recombined into a video, and then a target video may be obtained and output. Therefore, the source video may be output according to a target aspect ratio, and the playing effect of the output content may be ensured.

Based on the same concept, the embodiments of the disclosure further provide a video processing device.

It can be understood that in order to realize the above functions, the video processing device in the embodiments of the disclosure includes corresponding hardware structures and/or software modules for performing all functions. With reference to the units and algorithm steps of the examples disclosed in the embodiments of the disclosure, the embodiments of the disclosure may be implemented in a form of hardware or a combination of hardware and computer software. Whether a function is performed by hardware or computer software driving hardware depends on the specific applications and design constraints of the technical solutions. Those skilled in the art may use different methods to implement the described functions for each of the specific applications, but such implementation should not be considered to exceed the scope of the technical solutions of the embodiments of the disclosure.

<FIG> is a block diagram of a video processing device according to an embodiment. Referring to <FIG>, a video processing device <NUM> includes an extracting unit <NUM>, a clipping unit <NUM>, a scoring unit <NUM>, a selecting unit <NUM> and a generating unit <NUM>.

The extracting unit <NUM> is configured to extract a plurality of video frames from a source video with a first aspect ratio. The clipping unit <NUM> is configured to clip each of the plurality of video frames respectively multiple times to generate a plurality of candidate clipping boxes with a second aspect ratio. The scoring unit <NUM> is configured to respectively determine scores of the plurality of candidate clipping boxes of each of the plurality of video frames, and the scores represent playing effects of the video frames in response to that each of the candidate clipping boxes is taken as a clipping range. The selecting unit <NUM> is configured to select a clipping box of each of the video frames from the plurality of candidate clipping boxes based on the scores. The generating unit <NUM> is configured to clip each of the video frames by using the selected clipping box, and generate a target video with the second aspect ratio.

In an embodiment, the scoring unit <NUM> is configured to aesthetically score image contents in the candidate clipping boxes to obtain the scores of the candidate clipping boxes.

In another embodiment, the scoring unit <NUM> is configured to perform salient target detection for each of pixels in the candidate clipping boxes, and determine a pixel score of each of the pixels based on a salient target detection result; and determine the scores of the candidate clipping boxes according to the pixel score of each of the pixels in the candidate clipping boxes.

In another embodiment, the selecting unit <NUM> is configured, in response to selecting the clipping box of each of the video frames from the plurality of candidate clipping boxes based on the scores, to: construct a state transition matrix based on a position between every two candidate clipping boxes for each of the plurality of video frames, the state transition matrix being configured to smoothly control a jitter between adjacent frames for the selected candidate clipping box of each of the video frames; and determine a clipping box sequence which meets a maximum clipping box score by using the state transition matrix and the score of each of the candidate clipping boxes based on a maximum target function, the clipping box sequence including the selected candidate clipping box of each of the video frames.

In another embodiment, the selecting unit <NUM> is configured to construct the state transition matrix based on an IoU between every two candidate clipping boxes and a coefficient multiplier; and the coefficient multiplier is configured to control a degree of smoothing.

In another embodiment, the selecting unit <NUM> is configured to construct the state transition matrix based on a Euclidean distance between center positions of every two adjacent candidate clipping boxes, areas of the candidate clipping boxes and a coefficient multiplier; and the coefficient multiplier is configured to control a degree of smoothing.

In another embodiment, the video processing device <NUM> further includes an adjusting unit <NUM>, and the adjusting unit <NUM> is configured to: divide the source video into a plurality of video segments with an unchanged position and size of the selected clipping box between adjacent frames after the clipping box of each of the video frames is selected by the selecting unit <NUM>; determine a speed at which the clipping box will move in two adjacent video segments based on a middle time point of each of the two adjacent video segments and a distance between the clipping boxes in the two adjacent video segments; and adjust positions of the clipping boxes in the two adjacent video segments based on the speed and playing time.

In another embodiment, a total number of the plurality of video segments is K, and K is a positive integer greater than <NUM>.

The adjusting unit <NUM> is configured, in response to adjusting the positions of the clipping boxes in the two adjacent video segments based on the speed and the playing time, to: for a first video segment, in response to that the playing time is less than a middle time point of the first video segment, keep a position of the clipping box in the first video segment unchanged; for a kth video segment and a (k+<NUM>)th video segment, in response to that the playing time is greater than or equal to a middle time point of the kth video segment and less than a middle time point of the (k+<NUM>)th video segment, adjust the clipping box in the kth video segment to move a first distance; the first distance being a product of a first speed and a first time, the first speed being a speed at which the clipping box moves from the kth video segment to the (k+<NUM>)th video segment, the first time being a difference between a current time and the middle time point of the kth video segment, and k being a positive integer which meets the following conditions: <NUM><k<K-<NUM>; and for a Kth video segment, in response to that the playing time is greater than a middle time point of the Kth video segment, keep a position of the clipping box in the Kth video segment unchanged.

With respect to the device in the above embodiment, the specific manners for performing operations for individual modules therein have been described in detail in the embodiment regarding the method, which will not be elaborated herein.

<FIG> is a block diagram of a video processing device <NUM> according to the disclosure. For example, the device <NUM> may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a gaming console, a tablet, a medical device, exercise equipment, a personal digital assistant, and the like.

Referring to <FIG>, the device <NUM> may include one or more of the following components: a processing component <NUM>, a memory <NUM>, a power component <NUM>, a multimedia component <NUM>, an audio component <NUM>, an input/output (I/O) interface <NUM>, a sensor component <NUM> and a communication component <NUM>.

The processing component <NUM> usually controls overall operations of the device <NUM>, such as the operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing component <NUM> may include one or more processors <NUM> to execute instructions to complete all or part of the steps in the above method. Furthermore, the processing component <NUM> may include one or more modules which facilitate interaction between the processing component <NUM> and other components. For example, the processing component <NUM> may include a multimedia module to facilitate interaction between the multimedia component <NUM> and the processing component <NUM>.

Examples of such data include instructions for any applications or methods operated on the device <NUM>, contact data, phonebook data, messages, pictures, video, and the like. The memory <NUM> may be implemented by any type of volatile or non-volatile memory devices, or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, and a magnetic or optical disk.

The multimedia component <NUM> includes a screen providing an output interface between the device <NUM> and a user. The screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes the TP, the screen may be implemented as a touch screen to receive an input signal from the user. The TP includes one or more touch sensors to sense touches, swipes, and gestures on the TP. The touch sensors may not only sense a boundary of a touch or swipe action, but also detect a period of time and a pressure associated with the touch or swipe action. The multimedia component <NUM> includes a front camera and/or a rear camera. When the device <NUM> is in an operation mode, such as a photographing mode or a video mode, the front camera and/or the rear camera may receive external multimedia data. Each of the front camera and the rear camera may be a fixed optical lens system or have focusing and optical zooming capabilities.

The audio component <NUM> is configured to output and/or input an audio signal. For example, the audio component <NUM> includes a microphone (MIC), and the MIC is configured to receive an external audio signal when the device <NUM> is in an operation mode, such as a call mode, a recording mode and a voice recognition mode. The received audio signal may further be stored in the memory <NUM> or sent through the communication component <NUM>. The audio component <NUM> further includes a speaker configured to output the audio signal.

The buttons may include, but are not limited to: a home button, a volume button, a starting button, and a locking button.

The sensor component <NUM> includes one or more sensors configured to provide status assessments in various aspects for the device <NUM>. For example, the sensor component <NUM> may detect an on/off status of the device <NUM> and relative positioning of components, such as a display and small keyboard of the device <NUM>, and the sensor component <NUM> may also detect a change in a position of the device <NUM> or a component of the device <NUM>, presence or absence of contact between the user and the device <NUM>, orientation or acceleration/deceleration of the device <NUM>, and a change in temperature of the device <NUM>. The sensor component <NUM> may also include a light sensor, such as a complementary metal oxide semiconductor (CMOS) or charge coupled device (CCD) image sensor, configured for use in an imaging application. The sensor component <NUM> may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor or a temperature sensor.

The communication component <NUM> is configured to facilitate wired or wireless communication between the device <NUM> and other devices. The device <NUM> may access a communication-standard-based wireless network, such as a wireless fidelity (WiFi) network, a 2nd-generation (<NUM>) or 3rd-generation (<NUM>) network, or a combination thereof. The communication component <NUM> receives a broadcast signal or broadcast associated information from an external broadcast management system through a broadcast channel. The communication component <NUM> also includes a near field communication (NFC) module to facilitate short-range communications. For example, the NFC module may be implemented based on a radio frequency identification (RFID) technology, an infrared data association (IrDA) technology, an ultra-wide band (UWB) technology, a Bluetooth (BT) technology, and other technologies.

The device <NUM> may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors or other electronic components, and is configured to execute the above method.

In an embodiment, there is also provided a non-transitory computer-readable storage medium including instructions, such as included in the memory <NUM>, executable by the processor <NUM> of the device <NUM> to complete the above method. For example, the non-transitory computer-readable storage medium may be a ROM, a random access memory (RAM), a compact disc read-only memory (CD-ROM), a magnetic tape, a floppy disc, an optical data storage device, and the like.

It can be understood that in the disclosure, "a plurality of" refers to two or more, and other quantifiers are similar. "And/or" describes the relationships of associated objects, indicating that there may be three relationships. For example, A and/or B may indicate that there are three conditions: A exists alone, A and B exist simultaneously, and B exists alone. The character "/" generally indicates that the associated objects are in an "or" relationship. The singular forms "a", "said" and "the" are also intended to include multiple forms unless the context clearly indicates other meanings.

Further, it can be understood that the terms "first", "second", and the like are used to describe various types of information, but the information should not be limited to the terms. The terms are only used to distinguish the same type of information from each other, and do not indicate a specific order or degree of importance. In fact, the expressions such as "first" and "second" may be used interchangeably. For example, without departing from the scope of the disclosure, first information may also be referred to as second information. Similarly, second information may also be referred to as first information.

Claim 1:
A video processing method, the method comprising:
extracting (S11) a plurality of video frames from a source video with a first aspect ratio;
clipping (S12) each of the plurality of video frames respectively multiple times to generate a plurality of candidate clipping boxes with a second aspect ratio;
determining (S13) a score of each of the plurality of candidate clipping boxes of each of the plurality of video frames based on image contents in the candidate clipping box;
selecting (S14) a clipping box of each of the plurality of video frames from the plurality of candidate clipping boxes based on the scores of the plurality of candidate clipping boxes; and
clipping (S15) each of the plurality of video frames by using the selected clipping box, and generating a target video with the second aspect ratio,
wherein selecting the clipping box of each of the plurality of video frames from the plurality of candidate clipping boxes based on the scores of the plurality of candidate clipping boxes comprises:
constructing, for each of the plurality of video frames, a state transition matrix, wherein the state transition matrix is used for smoothing a jitter between adjacent video frames for the selected candidate clipping box of each of the plurality of video frames, characterized in that said step of constructing said state transition matrix is based on an intersection over union (IoU) between every two candidate clipping boxes and a coefficient multiplier, wherein the coefficient multiplier is used for controlling a degree of the smoothing; and,
establishing a maximum target function by using the state transition matrix and the score of each of the plurality of candidate clipping boxes, and determining a clipping box sequence which meets a maximum clipping box score based on the maximum target function, the clipping box sequence comprising the selected candidate clipping box of each of the plurality of video frames.