Patent Description:
In recent years, the way people acquire information and the way of entertainment have gradually changed, and the demand for video-type products has grown rapidly. If there is a phenomenon of picture jitter in a video, a user who is watching the video will suffer a poor experience.

Several common methods for detecting whether there is jitter in a video include: an optical flow method based on image displacement, a feature point matching method, and a video jitter detection method based on image gray distribution characteristics. Since video jitter will cause a certain degree of back and forth shaking of pictures, the optical flow method based on image displacement identifies the moving direction of the pictures through optical flows between image frames, thereby detecting whether a video is jittery. The feature point matching method identifies the moving direction of an object in an image by utilizing a feature point matching algorithm, thereby detecting whether a video is jittery. The video jitter detection method based on image gray distribution characteristics performs hypothesis testing in the row direction and in the column direction, respectively, by acquiring the expectation and variance of the row gray value and the expectation and variance of the column gray value, thereby detecting whether a video is jittery.

The above methods all perform detection on a video by utilizing image processing technology, and have complex calculation, lower accuracy and poorer robustness.

<CIT> discloses a method and a system for automatically identifying a shaking video, and the method includes: receiving an input video; and classifying the input video by using a classification model, where the classification model is obtained by taking a Flownet-Simple model as a basic network structure for training. The method overcomes the problems that the video jitter detection methods in the prior art only judge whether the video jitters through a displacement between local adjacent frames, easily make judgment errors in part of scenes (such as shot switching, video special effects), and lacks analysis on the whole video, which cause low detection accuracy.

The present invention provides a video jitter detection method and an apparatus, which improve the detection accuracy and robustness of video jitter.

In a first aspect, an embodiment of the present invention provides a video jitter detection method, including:.

Optionally, before inputting the video into the detection model, the method further includes:.

Optionally, the acquiring a first video from the video includes:.

The video sample set includes a first subset and a second subset, where video samples included in the first subset are labeled with evaluation values, and video samples included in the second subset are not labeled with evaluation values;
before inputting the video into the detection model, the method further includes:.

Optionally, the number of the video samples included in the first subset is less than the number of the video samples included in the second subset.

In a second aspect, an embodiment of the present invention provides a video jitter detection apparatus, including:.

Optionally, the acquiring module is further configured to:.

Optionally, the acquiring module is specifically configured to:.

The video sample set includes a first subset and a second subset, where video samples included in the first subset are labeled with evaluation values, and video samples included in the second subset are not labeled with evaluation values;
the apparatus further includes a training module, where the training module is configured to:.

Optionally, the training module is further configured to:.

In a third aspect, an embodiment of the present invention provides a video jitter detection apparatus, including: a memory and a processor; where.

In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, including: a readable storage medium and a computer program, where the computer program is used to implement the method provided by any implementation of the first aspect of the present invention.

The present invention provides a video jitter detection method and an apparatus. A video to be detected is input into a detection model, and an evaluation value of the video can be obtained through the detection model to determine whether the video is jittery. Since the detection model is a model trained according to a large number of samples, the video jitter detection method provided by the present invention has a wider range of application scenarios and improves the detection accuracy and robustness of video jitter.

In order to more clearly illustrate the technical solutions in embodiments of the present invention or in the prior art, the drawings required for the description of the embodiments or the prior art will be briefly described below. Obviously, the drawings in the following description are some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without any creative efforts.

In order to make the objects, technical solutions and advantages of embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described in combination with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative efforts fall within the protection scope of the present invention.

The terms "first", "second", "third", "fourth" and the like (if any) in the description and drawings of the present invention are used to distinguish similar objects, but are not necessarily used to describe a specific order or a precedence sequence. It should be understood that terms used in this way may be interchanged when appropriate, so that the embodiments of the present invention described herein can be implemented for example in other sequence in addition to those illustrated or described herein. Furthermore, the terms "include" and "have" and any of their variations are intended to cover non-exclusive inclusions, for example, processes, methods, systems, products, or devices that include a series of steps or units are not necessarily limited to those steps or units listed explicitly, instead may include other steps or units that are not explicitly listed or that are inherent in these processes, methods, products or devices.

The technical solutions of the present invention will be illustrated in detail in the following specific embodiments. The following specific embodiments may be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

<FIG> is a flowchart of a video jitter detection method according to an embodiment of the present invention. The executive entity of the video jitter detection method provided by this embodiment may be a video jitter detection apparatus. As shown in <FIG>, the video jitter detection method provided by this embodiment includes:.

The detection model is a model obtained by training using video samples in a video sample set as inputs, and evaluation values of the video samples in the video sample set as outputs.

Specifically, a video to be detected is acquired, and an evaluation value corresponding to the video to be detected is obtained end-to-end through the detection model, and then it can be determined according to the evaluation value of the video whether there is a phenomenon of jitter in the video. Since the detection model is an end-to-end model obtained by training using a large number of video samples as inputs, and the evaluation values corresponding to the video samples as outputs, good judgment effect on whether there is video jitter in the video and on the degree of video jitter when the video jitter exists is realized. Compared with the video jitter detection based on image processing technology in the prior art, the calculation complexity is reduced, the range of application is expanded with the applicability to the detection of various videos, and the detection accuracy and robustness of video jitter are improved.

It should be noted that the size, file format, video length and the like of the acquired video are not limited in this embodiment.

It should be noted that the implementation of the evaluation value is not limited in this embodiment. Optionally, the evaluation value may include a limited number of discrete values. For example, the evaluation value may include: no jitter, and jitter. The evaluation value may include: no jitter, slight jitter, and severe jitter. The evaluation value may include: <NUM> and <NUM>, where <NUM> represents that there is no jitter, and <NUM> represents that there is jitter. The evaluation value may include: <NUM>, <NUM>, and <NUM>, where <NUM> represents that there is no jitter, <NUM> represents that there is slight jitter, and <NUM> represents that there is severe jitter. Optionally, the evaluation value may be a numerical value in a continuous range. For example, the value range of the evaluation value may be greater than or equal to <NUM> and less than or equal to <NUM>. The greater the evaluation value is, the more jittery the video is. For example, the jitter of a video with an evaluation value of <NUM> is more severe than that of a video with an evaluation value of <NUM>. Or, the smaller the evaluation value is, the more jittery the video is. The value range of the evaluation value is not limited in this embodiment. For example, the value range of the evaluation value may be greater than or equal to <NUM> and less than or equal to <NUM>. Optionally, when the evaluation value has a certain value range, a preset numerical value can be set. The preset numerical value is used to determine whether the video is jittery. For example, the value range of the evaluation value is greater than or equal to <NUM> and less than or equal to <NUM>, and the greater the evaluation value is, the more jittery the video is. The preset numerical value may be <NUM>. When the evaluation value of the video is greater than <NUM>, it is determined that the video is jittery, and when the evaluation value of the video is less than or equal to <NUM>, it is determined that the video is not jittery. The value of the preset numerical value is not limited in this embodiment.

It should be noted that, the number of the video samples included in the video sample set is not limited in this embodiment. In order to improve the accuracy of the detection model and solve the problem of overfitting during training, the number of the video samples is as large as possible. For example, the number of the video samples may be greater than <NUM>,<NUM>, such as <NUM>,<NUM>.

It should be noted that, the type of the detection model is not limited in this embodiment. Optionally, the type of the detection model may be any one of the following: a neural network model, a depth algorithm model and a machine algorithm model. Optionally, the neural network model may include, but is not limited to, a fully convolutional network (FCN) model and a convolutional network for biomedical image segmentation (U-net) model.

Optionally, in S102, before inputting the video into the detection model, the method further includes:.

Correspondingly, in S102, the inputting the video into the detection model can include:
inputting the first video into the detection model.

Specifically, if the length of the video to be detected is too long, the time of detection will be increased. Moreover, the shooting environment of the video is usually constant, so if the video to be detected is too long, it can be determined whether the video is jittery by detecting a part of the video, and it is generally not necessary to perform detection on the whole video. Therefore, when the duration of the video to be detected is greater than the preset duration, the first video with the preset duration is acquired from the video to be detected, and the first video is input into the detection model to determine whether the video is jittery, which reduces the amount of calculation, shortens the time of detection and improves the detection efficiency.

It should be noted that, the value of the preset duration is not limited in this embodiment. For example, the preset duration may be <NUM> seconds.

Optionally, in an implementation, the acquiring the first video from the video may include:
acquiring, in the video, the first video from a starting time of the video.

For example, assuming that the preset duration is <NUM> seconds, and the duration of the video to be detected is <NUM> minutes, then the first <NUM> seconds of the video to be detected is determined as the first video.

Usually, the user is used to watching the video from beginning after clicking on the video. If there is jitter at the beginning of the video, the user's experience when watching the video is directly affected. Therefore, the acquiring the first video from the starting time of the video takes the watching habit of the user into account, which improves the accuracy and effectiveness of video detection, and is closer to the user experience.

Optionally, in another implementation, the acquiring the first video from the video may include:
acquiring, from the video, multiple segments of short videos with time intervals therebetween, connecting the multiple segments of short videos and determining the connected video as the first video.

For example, assuming that the preset duration is <NUM> seconds and the duration of the video to be detected is <NUM> minutes, then the first video may include the following time periods in the video to be detected: <NUM>-<NUM> seconds, <NUM>-<NUM> seconds, and <NUM>-<NUM> seconds, or include <NUM>-<NUM> seconds, and <NUM>-<NUM> seconds.

Usually, the user is used to watching the video from beginning after clicking on the video, and the user may jump to different parts of the video for multiple times during the watching. Therefore, determining the multiple segments of short videos in the video as the first video considers the watching habit of the user, which improves the accuracy and effectiveness of video detection, and is closer to the user experience.

It should be noted that, the number and duration of the short videos, and their positions in the video to be detected are not limited in this embodiment.

In S102, before inputting the video into the detection model, the method further includes:
training the detection model.

It should be noted that, the training method of the detection model is not limited in this embodiment, and may vary according to different types of the model. For example, during the training, frame-cut processing may be performed on the video with <NUM> frames per second, and the video frame is scaled to the size of <NUM>*<NUM> in the format of a three-primary-color (Red, Green, Blue, RGB) image.

It can be seen that, this embodiment provides a video jitter detection method, in which the video to be detected is input into the detection model, and the evaluation value of the video can be obtained through the detection model to determine whether the video is jittery. Since the detection model is the model trained according to a large number of samples, the video jitter detection method provided by this embodiment has a wider range of application scenarios, and improves the detection accuracy and robustness of video jitter.

<FIG> is another flowchart of a video jitter detection method according to an embodiment of the present invention. A training process of the detection model is illustrated in this embodiment. In this embodiment, the video sample set includes a first subset and a second subset. Video samples included in the first subset are labeled with evaluation values, and video samples included in the second subset are not labeled with evaluation values. As shown in <FIG>, the training the detection model includes:.

Illustration is given in combination with examples in the following.

Assuming that the video sample set includes <NUM>,<NUM> video samples in total, where the first subset includes <NUM>,<NUM> video samples, and these <NUM> video samples are labeled with evaluation values. The acquisition of the evaluation values of the video samples in the first subset is not limited in this embodiment, for example, they may be acquired through manually labeling. The second subset includes <NUM>,<NUM> video samples, and these <NUM>,<NUM> video samples are not labeled with evaluation values.

The training process of the detection model can include that:
the first model is obtained by training using the <NUM> video samples in the first subset as inputs, and the evaluation values respectively corresponding to the <NUM> video samples as outputs. Then, the <NUM>,<NUM> video samples in the second subset are input into the first model to obtain evaluation values respectively corresponding to the <NUM>,<NUM> video samples. Then, the modified second subset corresponding to the second subset is acquired. Specifically, through manual correction, the evaluation values are manually corrected for the video samples, whose evaluation values are greater than the first numerical value, in the second subset. For convenience of explanation, for the evaluation values of the video samples in the second subset, the evaluation values obtained through the first model are denoted as evaluation values P1, and the evaluation values obtained through the manual correction are denoted as evaluation values P2. For example, the first numerical value is <NUM>. The evaluation value P1 of a video sample <NUM> in the second subset is <NUM>. The evaluation value P2 of the video sample <NUM> is obtained by performing manual correction for the video sample <NUM>, and may be <NUM>. Finally, the detection model is obtained by training using the <NUM> video samples in the first subset and the <NUM>,<NUM> video samples in the modified second subset as inputs, and the evaluation value of each video sample as the output.

It can be seen that, the first model is obtained by training using a part of video samples in the video sample set, and the accuracy rate needs to be improved. The video samples included in the second subset are not labeled with evaluation values. First, the evaluation value of each video sample can be obtained through the first model. Since there may be an error in the first model, manual correction is performed on video samples with higher evaluation values output by the first model. Then, according to the video samples in the first subset and the modified second subset, training is continued based on the first model, to obtain an accurate detection model.

S203 to S204 can be executed in loop for multiple times, and the number of execution is not limited in this embodiment.

The value of the first numerical value is not limited in this embodiment.

Optionally, the number of the video samples included in the first subset may be less than the number of the video samples included in the second subset, which reduces the difficulty of acquiring the first subset.

Optionally, after S204, the method further includes:.

Illustration is given in combination with the above examples in the following.

Assuming that the third subset includes <NUM>,<NUM> video samples, and these <NUM>,<NUM> video samples are not labeled with evaluation values.

The training process of the detection model can also include that:
the <NUM>,<NUM> video samples in the third subset are input into the detection model, to obtain evaluation values respectively corresponding to the <NUM>,<NUM> video samples. The check samples are acquired from the third subset. Specifically, manual correction is performed on the video samples, whose evaluation values obtained using the detection model are greater than the second numerical value, in the third subset. For convenience of explanation, for the evaluation values of the video samples in the third subset, the evaluation values obtained through the detection model are denoted as evaluation values Q1, and the evaluation values obtained through the manual correction are denoted as evaluation values Q2. For example, the second numerical value is <NUM>. The evaluation value Q1 of a video sample <NUM> in the third subset is <NUM>. The evaluation value Q2 of the video sample <NUM> is obtained by performing manual correction on the video sample <NUM>, and may be <NUM>. The video sample <NUM> is a check sample. Assuming that the number of the check samples is <NUM>. Finally, the detection model is trained using the <NUM> video samples in the first subset, the <NUM>,<NUM> video samples in the modified second subset and the <NUM> check samples as inputs, and the evaluation value of each video sample as the output, so as to correct the detection model.

It can be seen that, the accuracy of the detection model is further improved by performing correction on the detection model through the third subset.

<FIG> is a schematic structural diagram of a video jitter detection apparatus according to an embodiment of the present invention. The video jitter detection apparatus provided by this embodiment is used to execute the video jitter detection methods provided by the embodiments shown in <FIG>. As shown in <FIG>, the video jitter detection apparatus provided by this embodiment includes:.

Optionally, the acquiring module <NUM> is further configured to:.

Optionally, the acquiring module <NUM> is specifically configured to:.

The video sample set includes a first subset and a second subset, where the video samples included in the first subset are labeled with evaluation values, and the video samples included in the second subset are not labeled with evaluation values;
the apparatus further includes a training module, which is configured to:.

Optionally, the number of the video samples included in the first subset is smaller than the number of the video samples included in the second subset.

The video jitter detection apparatus provided by this embodiment is used to execute the video jitter detection methods provided by the embodiments shown in <FIG>. The technical principles and technical effects thereof are similar, and will not be repeated here.

<FIG> is another schematic structural diagram of a video jitter detection apparatus according to an embodiment of the present invention. As shown in <FIG>, the video jitter detection apparatus can include a processor <NUM> and a memory <NUM>. The memory <NUM> is configured to store instructions, and the processor <NUM> is configured to execute the instructions stored in the memory <NUM> to cause the video jitter detection apparatus to execute the video jitter detection methods provided by the embodiments shown in <FIG>. The technical principles and technical effects thereof are similar, and will not be repeated here.

It should be noted that the present invention does not limit the device form and specific structure of the video jitter detection apparatus.

Exemplary, <FIG> is a schematic structural diagram of an exemplary computer system/server according to an embodiment of the present invention, where the computer system/server is used for implementing embodiments of the present invention. The computer system/server <NUM> shown in <FIG> is only an example, and should not impose any limitation on the function and usage range of the embodiments of the present invention.

As shown in <FIG>, the computer system/server <NUM> is represented in a form of a general-purpose computing device. Components of the computer system/server <NUM> may include, but are not limited to: one or more processors or processor units <NUM>, a system memory <NUM>, and a bus <NUM> connecting different system components (including the system memory <NUM> and the processors <NUM>).

The bus <NUM> represents one or more of several types of bus structures, including a memory bus or a memory controller, a peripheral bus, a graphic acceleration port, a processor, or a local bus using any of multiple bus structures. By way of example, these architectures include, but are not limited to, an industry standard architecture (ISA) bus, a micro channel architecture (MAC) bus, an enhanced ISA bus, a video electronics standards association (VESA) local bus, and a peripheral component interconnect (PCI) bus.

The computer system/server <NUM> typically includes a variety of computer system readable media. These media may be any available medium that can be accessed by the computer system/server <NUM>, including volatile and non-volatile medium, removable and non-removable medium.

The system memory <NUM> may include computer system readable medium in a form of volatile memory, such as a random access memory (RAM) <NUM> and/or a cache memory <NUM>. The computer system/server <NUM> may further include other removable/non-removable, volatile/nonvolatile computer system storage medium. For example only, a storage system <NUM> may be used to read and write on a non-removable, non-volatile magnetic medium (which is not shown in <FIG> and is commonly referred as a "hard driver"). Although not shown in <FIG>, a disk drive for reading and writing on a removable non-volatile disk (for example, a "floppy disk") and an optical drive for reading and writing on a removable non-volatile optical disk (for example, a compact disc read - only memory (CD-ROM), a digital video disk - ROM (DVD-ROM) or other optical medium) may be provided. In these cases, each drive may be connected to the bus <NUM> through one or more data medium interfaces. The memory <NUM> may include at least one program product, which has a set of (for example, at least one) program modules, and these program modules are configured to implement functions of various embodiments of the present invention.

A program/utility tool <NUM> having a set of (at least one) program modules <NUM> may be stored in, for example, the memory <NUM>. Such program modules <NUM> include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each of these examples, or a certain combination thereof, may include an implementation of a network environment. The program modules <NUM> generally implement functions and/or methods in the embodiments described in the present invention.

The computer system/server <NUM> may also communicate with one or more peripheral devices <NUM> (for example, a keyboard, a pointing device, a display <NUM>, and the like), and may also communicate with one or more devices that enable a user to interact with the computer system/server <NUM>, and/or communicate with any device (such as a network card, a modem, and the like) that enables the computer system/server <NUM> to communicate with one or more other computing devices. Such communication may be performed through an input/output (I/O) interface <NUM>. Moreover, the computer system/server <NUM> may also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and/or a public network, such as Internet) through a network adapter <NUM>. As shown in <FIG>, the network adapter <NUM> communicates with other modules of the computer system/server <NUM> through the bus <NUM>. It should be understood that although not shown in <FIG>, other hardware and/or software modules may be used in combination with the computer system/server <NUM>, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, redundant array of independent disks (RAID) systems, tape drives, data backup storage systems, and the like.

The processor <NUM> executes the functions and/or methods in the embodiments described in the present invention by running the program stored in the system memory <NUM>.

The above-mentioned computer program may be provided in a computer storage medium, that is, the computer storage medium is encoded with a computer program, and when executed by one or more computers, the program causes the one or more computers to execute the method flows and/or apparatus operations shown in the foregoing embodiments of the present invention.

A person of ordinary skill in the art may understand that: the implementation of all or part of the steps of the foregoing various method embodiments may be completed by hardware related to program instructions. The aforementioned program may be stored in a computer readable storage medium. When the program is executed, the steps including the foregoing various method embodiments are executed; and the foregoing storage medium includes various media that can store program codes, such as a ROM, a RAM, a magnetic disk, or an optical disc.

Claim 1:
A video jitter detection method, comprising:
acquiring (<NUM>) a video;
inputting (<NUM>) the video into a detection model to obtain an evaluation value of the video, wherein the evaluation value is used to indicate a degree of jitter of the video; wherein the detection model is a model obtained by training using video samples in a video sample set as inputs, and evaluation values of the video samples in the video sample set as outputs;
wherein the video sample set comprises a first subset and a second subset, wherein video samples comprised in the first subset are labeled with evaluation values, video samples comprised in the second subset are not labeled with evaluation values,
and training the detection model is obtained by the following manner:
obtaining (<NUM>) a first model by training using the video samples in the first subset as inputs, and the evaluation values of the video samples in the first subset as outputs;
inputting (<NUM>) the video samples in the second subset into the first model to obtain evaluation values of the video samples in the second subset;
acquiring (<NUM>) a modified second subset by manually correcting evaluation values for video samples, whose evaluation values are greater than a first numerical value, in the second subset, wherein the video samples in the modified second subset are labeled with evaluation values; and
obtaining (<NUM>) the detection model by training using the video samples in the first subset and the video samples in the modified second subset as inputs, and the evaluation values respectively corresponding to the video samples in the first subset and the video samples in the modified second subset as outputs.