Patent Description:
The aspect ratio of an image is a proportional relationship between a width of the image and a height of the image. Various aspect ratios have been introduced for different exhibition and broadcast media, such as the standard <NUM>:<NUM> aspect ratio for television viewing and the <NUM>:<NUM>:<NUM> widescreen aspect ratio for viewing in cinemas. Black bars may be added to the image frames of a video, on the top and bottom and/or sides, in order to transfer the video from one aspect ratio to another while preserving the original aspect ratio. For example, letterboxing has been used to transfer widescreen aspect ratio videos to standard-width video formats by adding black bars to the top and bottom of the widescreen aspect ratio videos.

When black bars are added to the image frames of a video, aspect ratio errors may be introduced such that non-black regions of the image frames do not maintain a desired aspect ratio. For example, a row (or column) of pixels in an image frame may become part of the surrounding black bars instead of the desired image, or the non-black regions in the image frames of a video may suddenly switch from one aspect ratio to another. Manual identification of such aspect ratio errors is a time consuming process that can cause eye fatigue and attention strain, and humans can also miss aspect ratio errors in videos. Traditional automated tools for checking aspect ratio errors have produced many false positives, by improperly identifying aspect ratio errors.

Document <CIT> describes playback control methods and systems. At least one picture is obtained. The picture corresponds to a frame, and the picture includes a first region and a second region adjacent to the first region. It is determined whether at least one pixel in the first region has a predefined color, and whether at least one pixel in the second region has the predefined color. When the pixel in the first region has the predefined color and the pixel in the second region does not have the predefined color, the first region is removed, and the second region is enlarged to cover an original position of the first region. The enlarged second region is then played back via an anamorphic lens.

Document <CIT> describes systems and method for identifying a video aspect-ratio frame attribute of a current frame. One example embodiment takes the form of a method including: (i) receiving the current frame; (ii) defining a region of the current frame, the defined region having a plurality of lumas within a luma range; (iii) calculating a non-black luma percentage of the region based on the lumas; (iv) calculating an average luma of the region based on the lumas; and (v) identifying the current frame as having a standard video aspect-ratio attribute responsive to (a) the average luma being less than a max-black luma threshold and (b) the non-black luma percentage being less than a non-black luma percentage threshold.

The current invention includes a computer-implemented method for recognizing aspect ratio changes that are indicative of aspect ratio errors in a video including a plurality of image frames, according to claim <NUM>. The method includes, determining, for each image frame of the plurality of image frames, positions of a top, a bottom, a left, and a right of a content region in the image frame, wherein the content region is the image frame exclusive of any black bars therein that have color values within a threshold from a color value for black. The method further includes generating at least one time series from the determined positions. In addition, the method includes determining aspect ratio changes based on calculating only the aspect ratios of content regions in image frames of the plurality of image frames corresponding to non-zero points in at least one first differential of the at least one time series.

According to one embodiment, the positions of the top, the bottom, the left, and the right of the content region in each image frame of the plurality of image frames are determined as distances from a top edge, a bottom edge, a left edge, and a right edge of the image frame, respectively.

According to one embodiment, determining, for each image frame of the plurality of image frames, the positions of the top, the bottom, the left, and the right of the content region in the image frame includes: skipping completely black rows at the top and the bottom of the image frame until the top and the bottom, respectively, of the content region are reached; rotating the image frame by ninety degrees; and skipping completely black rows at a top and a bottom of the rotated image frame until the left and the right, respectively, of the content region are reached.

According to one embodiment, all pixels in the completely black rows have color values within a threshold from a color value for black.

According to one embodiment, the method further comprises determining that an image frame of the plurality of image frames does not include the content region based on a determination that all pixels of the image frame have color values within a threshold from a color value for black.

According to one embodiment, the method further comprises reporting, as an aspect ratio error, the aspect ratio change and associated image frames in the video.

According to one embodiment, the at least one time series includes either (i) a time series that includes the positions of the top, the bottom, the left, and the right of the content regions in the plurality of image frames, or (ii) distinct time series for the positions of each of the top, the bottom, the left, and the right of the content regions in the plurality of image frames.

According to one embodiment, multiple chunks of the plurality of image frames from the video are processed in parallel.

According to one embodiment, the method further comprises splicing the video into the plurality of image frames.

The invention also includes a non-transitory computer-readable storage medium according to claim <NUM>.

More specifically, according to one aspect a non-transitory computer-readable storage medium includes instructions that, when executed by a processing unit, cause the processing unit to recognize aspect ratio errors in a video including a plurality of image frames, by performing operations comprising: determining, for each image frame of the plurality of image frames, positions of a top, a bottom, a left, and a right of a content region in the image frame, wherein the content region is the image frame exclusive of any black bars therein that have color values within a threshold from a color value for black, generating at least one time series from the determined positions; and determining aspect ratio changes based on calculating only the aspect ratios of content regions in images frames of the plurality of image frames corresponding to non-zero points in at least one first differential of the at least one time series.

According to one aspect, the positions of the top, the bottom, the left, and the right of the content region in each image frame of the plurality of image frames are determined as distances from a top edge, a bottom edge, a left edge, and a right edge of the image frame, respectively.

According to one aspect, the determining, for each image frame of the plurality of image frames, the positions of the top, bottom, left, and right of the content region in the image frame includes: skipping completely black rows at the top and bottom of the image frame until the top and the bottom, respectively, of the content region are reached;
rotating the image frame by ninety degrees; and skipping completely black rows at the top and bottom of the rotated image frame until the left and right, respectively, of the content region are reached.

According to one aspect, the completely black rows include pixels whose values are less than a predefined threshold from a value for black.

According to one aspect, the operations further comprise determining that a first image frame of the plurality of image frames does not include the content region based on a determination that all pixels of the first image frame have color values within a threshold from a color value for black.

According to one aspect, the operations further comprise reporting, via a display device, the determined aspect ratios and associated image frames in the video.

According to one aspect, the at least one time series includes either a time series that includes the determined positions of top, bottom, left, and right of the content regions in the plurality of image frames or distinct time series for positions of each of the top, bottom, left, and right of the content regions in the plurality of image frames.

According to one aspect, multiple chunks of the plurality of image frames from the video are processed in parallel.

According to one aspect, wherein determining, for each image frame of the plurality of image frames, the positions of the top and the bottom of the content region in the image frame comprises skipping completely black rows at the top and the bottom of the image frame until the top and the bottom, respectively, of the content region are reached.

The invention also includes a system according to claim <NUM>.

More specifically, according to one aspect a system, comprises: a computer processor; and a memory containing a program that, when executed on the computer processor, performs operations for recognizing aspect ratio changes that are indicative of aspect ratio errors in a video including a plurality of image frames, the operations comprising: determining, for each image frame of the plurality of image frames, positions of a top, bottom, left, and right of a content region in the image frame wherein the content region is the image frame exclusive of any black bars therein that have color values within a threshold from a color value for black; generating at least one time series from the determined positions; and determining aspect ratio changes based on calculating only the aspect ratios of content regions in image frames of the plurality of image frames corresponding to non-zero points in at least one first differential of the at least one time series.

According to one embodiment, the positions of the top, bottom, left, and right of the content region in each image frame of the plurality of image frames are determined as distances from top, bottom, left, and right edges of the image frame, respectively.

According to one embodiment, the operations further comprise determining that an image frame of the plurality of image frames does not include the content region based on a determination that all pixels of the image frame have color values within a threshold from a color value for black.

In an illustrative implementation, the operations may further comprise reporting, via a display device, the determined aspect ratios and associated image frames in the video.

In an illustrative implementation, the operations may further comprise splicing the video into the plurality of image frames.

So that the manner in which the above recited aspects are attained and can be understood in detail, a more particular description of aspects of this disclosure, briefly summarized above, may be had by reference to the appended drawings.

It is to be noted, however, that the appended drawings illustrate only typical aspects of this disclosure and are therefore not to be considered limiting of its scope, for the disclosure may admit to other equally effective configurations.

Aspects of this disclosure provide an approach for recognizing aspect ratio errors in image frames of a video and reporting the same. In one configuration, an aspect ratio checker application receives a video that includes multiple image frames and identifies aspect ratio changes in those image frames based on a first differential of a time series that includes positions of the top, bottom, left, and right of content regions within the image frames. As used herein, a "content region" refers to an image frame region that remains after black bars have been removed from the image frame, i.e., the image frame exclusive of any black bars therein. The aspect ratio checker is configured to compute a first differential of the aforementioned time series, identify aspect ratio changes based on points where the first differential is non-zero, and determine the aspect ratio of content regions within image frames corresponding to the non-zero points of the first differential. The aspect ratio checker may further generate and display a report indicating the determined aspect ratios and image frames ranges associated with those aspect ratios.

In the following, reference is made to aspects of the present disclosure. However, it should be understood that this disclosure is not limited to specific described aspects. Instead, any combination of the following features and elements, whether related to different aspects or not, is contemplated to implement and practice the aspects disclosed herein. Furthermore, although aspects may achieve advantages over other possible solutions and/or over the prior art, whether or not a particular advantage is achieved by a given aspect is not limiting of this disclosure. Thus, the following aspects, features, configurations and advantages are merely illustrative and are not considered elements or limitations of the appended claims except where explicitly recited in a claim(s). Likewise, reference to "the aspect" shall not be construed as a generalization of any inventive subject matter disclosed herein and shall not be considered to be an element or limitation of the appended claims except where explicitly recited in a claim(s).

As will be appreciated by one skilled in the art, aspects of the present disclosure may be embodied as a system, method or computer program product. Accordingly, aspects of the present disclosure may take the form of an entirely hardware configuration, an entirely software configuration (including firmware, resident software, micro-code, etc.) or a configuration combining software and hardware aspects that may all generally be referred to herein as a "circuit," "module" or "system. " Furthermore, aspects of the present disclosure may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied thereon.

Aspects of the present disclosure are described below with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to aspects of the disclosure.

Aspects of the present disclosure may be provided to end users through a cloud computing infrastructure. Cloud computing generally refers to the provision of scalable computing resources as a service over a network. More formally, cloud computing may be defined as a computing capability that provides an abstraction between the computing resource and its underlying technical architecture (e.g., servers, storage, networks), enabling convenient, on-demand network access to a shared pool of configurable computing resources that can be rapidly provisioned and released with minimal management effort or service provider interaction. Thus, cloud computing allows a user to access virtual computing resources (e.g., storage, data, applications, and even complete virtualized computing systems) in "the cloud," without regard for the underlying physical systems (or locations of those systems) used to provide the computing resources.

Typically, cloud computing resources are provided to a user on a pay-per-use basis, where users are charged only for the computing resources actually used (e.g. an amount of storage space consumed by a user or a number of virtualized systems instantiated by the user). A user can access any of the resources that reside in the cloud at any time, and from anywhere across the Internet. In context of the present disclosure, a user may access applications (e.g., an aspect ratio checker application) or related data available in the cloud. For example, the aspect ratio checker could execute on a computing system in the cloud, determine aspect ratio errors in the image frames of a video according to techniques disclosed herein, and store the results of the determination at a storage location in the cloud. Doing so allows a user to access this information from any computing system attached to a network connected to the cloud (e.g., the Internet).

Referring now to <FIG>, an approach is illustrated for detecting and reporting aspect ratio errors in image frames of a video, according to an aspect of this disclosure. As shown, an aspect ratio checker application <NUM> receives as input a video <NUM> that includes multiple image frames. In one configuration, the aspect ratio checker <NUM> may (<NUM>) splice the received video <NUM> into its component image frames <NUM>, (<NUM>) determine whether any of the image frames <NUM> are all black, (<NUM>) determine, for each of the image frames <NUM> that is not all black, positions of the top, bottom, left, and right of a content area within the image frame, (<NUM>) generate a time series <NUM> from the determined positions, (<NUM>) determine a first differential <NUM> of the time series <NUM>, (<NUM>) determine aspect ratios of content regions in image frames that correspond to points where the first differential <NUM> is non-zero, and (<NUM>) output a report indicating, e.g., the determined aspect ratios and image frame ranges associated with those aspect ratios.

Subsequent to splicing the video <NUM> into the image frames <NUM>, the aspect ratio checker <NUM> determines whether any of the image frames <NUM> are completely black. The aspect ratio checker <NUM> disregards completely black image frames when checking for aspect ratio errors, as it is unnecessary and indeed impossible to check the aspect ratio of a (non-existent) content region within a black image frame. In one configuration, each image frame may be represented as a matrix of pixels in which each pixel is represented by three values on the RGB (red, green, blue) spectrum. When determining whether an image frame is all black, differences between pixel color values and a value of the color black may be tolerated up to a threshold of tolerance. Doing so may reduce the number of false positives that would otherwise be produced if a more exact numerical match were required. For example, in a color model where a color's value can range from <NUM> to <NUM>, with <NUM> being true black, the threshold may be set to between <NUM> and <NUM>.

Subsequent to identifying and skipping completely black image frames, the aspect ratio checker <NUM> determines, for each remaining image frame (that is not all black), positions of the top, bottom, left, and right of a content area within the image frame. In one configuration, the determined positions may be represented as (and/or converted into) distances from the top, bottom, left, and right edges of the image frame, as opposed to (x,y) coordinates in the image frames. <FIG> illustrates an example of determining positions of the top, bottom, left, and right of a content area <NUM> within an image frame <NUM>, according to an aspect of this disclosure. As shown in panel A, the aspect ratio checker <NUM> determines a position of the top of the content region <NUM> by starting from a top row of pixels <NUM> of the image frame <NUM> and proceeding downward, checking every row to determine whether the row includes all black pixels, and breaking when a row <NUM> of pixels that is not all black is reached, indicating the top of the content region <NUM>. To check for rows of black pixels, the aspect ratio checker <NUM> may compare the image frame <NUM> on a row-by-row and (within each row a) pixel-by-pixel basis to a value of the color black and identify rows having pixel values that are all within a threshold of tolerance of the black color value, such between <NUM> and <NUM> where the color of black is <NUM>. The aspect ratio checker <NUM> may likewise determine a position of the bottom of the content region <NUM> by starting from a bottom row <NUM> of the image frame <NUM> and proceeding upward until a row <NUM> that is not all black is reached, indicating the bottom of the content region <NUM>.

The foregoing process may be repeated to determine positions of the left and right sides of the content region <NUM> after the aspect ratio checker <NUM> rotates the image frame <NUM> by <NUM> degrees, as shown in panel B. Illustratively, the aspect ratio checker <NUM> determines a left side of the content region <NUM> by starting from a top row <NUM> of the rotated image frame <NUM> (after the image frame <NUM> is rotated by <NUM> degrees) and proceeding downward to check for rows that are not all black. In this example, the top row <NUM> of the rotated image frame <NUM> is not all black, so the aspect ratio checker <NUM> immediately breaks and stores a position of the top row as the left of the content region <NUM>. However, in frames that contain vertical letterboxing, the aspect ratio checker <NUM> may not identify the left of the content region <NUM> immediately starting from the top of the rotated image frame <NUM>. The aspect ratio checker <NUM> may similarly determine a position of the right side of the content region <NUM> by starting from a bottom row <NUM> of the rotated image frame <NUM> and proceeding upward until a row that is not all black is reached (in this case, the bottom row <NUM> is not all black), indicating the right side of the content region <NUM>.

In one configuration, the aspect ratio checker <NUM> may determine coordinates of the top of the content region <NUM> that the aspect ratio checker <NUM> then normalizes to a distance from the top edge of the image frame, and similarly for the bottom, left, and right of the content region <NUM>. That is, the determined positions of the top, bottom, left, and right of the content region <NUM> may initially be in the form of coordinates, such as (x,y) coordinates with respect to the entire image frame, and the aspect ratio checker <NUM> may normalize such coordinates so that the positions of the top, bottom, left, and right of the content region <NUM> are expressed in terms of distances from corresponding edges of the image frame. The following code may be used to perform such normalization in a particular configuration of the aspect ratio checker <NUM>: top_distance = top; bottom_distance = frame_height - bottom; left_distance = left; right_distance = frame-width - right.

Although a <NUM> degree rotation is used in the example of <FIG>, it should be understood that alternative configurations of the aspect ratio checker <NUM> may determine positions of the top, bottom, left, and right of content regions in other ways, such as by proceeding from the left and right of image frames to determine the left and right positions of content regions, rather than rotating by <NUM> degrees and proceeding from the top and bottom of the image frames. Further, although the example image frame <NUM> in <FIG> includes black bars on the top and bottom, but not the sides of the image frame <NUM>, techniques described herein for determining positions of the top, bottom, left, and right of content regions within the image frames of a video are also applicable to image frames with black bars on only on the sides, or on both the top and bottom and the sides.

Returning to <FIG>, after determining positions of the top, bottom, left, and right of the content region in each image frame, the aspect ratio checker <NUM> generates a time series <NUM> based on the determined positions and determines a first differential <NUM> of the time series <NUM>. For example, the time series <NUM> may be an array indicating image frame numbers, as well as the determined positions of the top, bottom, left, and right of content regions within those image frames. Although discussed herein primarily with respect to a single time series that includes the top, bottom, left, and right positions of content regions, which may be expressed as distances from the (top, bottom, left, and right) edges of each image frame, in an alternative configuration, the top, bottom, left, and right edge distances may be separated into distinct time series. In such a case, each of the distinct time series may be processed (by taking a first differential, etc.) in a similar manner as the time series with combined top, bottom, left, and right edge distances. That is, either one time series may be created that includes the top, bottom, left, and right position information and a first differential taken of such a time series, after which the top, bottom, left, and right components separated out thereafter, or the distinct time series may be created for each of the top, bottom, left, and right positions in the beginning.

<FIG> illustrates an example plot of a time series that includes positions of the top, bottom, left, and right of content regions within the image frames of a video, expressed as edge distances, as well as a first differential of the time series. As shown in panel A, the time series indicates that the distance from a corresponding edge (in pixels) of one of the top, bottom, left, and right of the content regions increases at time <NUM> and decreases back to the original edge distance at a later time <NUM>. Other edge distances do not change from time <NUM> to time <NUM>, as shown by the (overlapping) edge distances <NUM> that remain constant. For example, the top (or bottom, left, or right) of content regions within the image frames may suddenly move downward (or upward, to the right, or to the left where the bottom, left, or right of the content regions change positions), increasing the edge distance at time <NUM>, and vice versa at time <NUM>. Similarly, the edge distance of one of the top, bottom, left, and right of the content regions within the image frames increases again at time <NUM> and decreases back to the original edge distance at time <NUM>, while other edge distances <NUM> remain unchanged.

Panel B shows the first differential of the edge distances shown in panel A. As shown, the first differential of edge distances is zero everywhere except at points <NUM>, <NUM>, <NUM>, and <NUM>, which correspond to the times <NUM>, <NUM>, <NUM>, and <NUM>, respectively, where the edge distances of one of the top, bottom, left, and right of the content regions changes. The first differential may generally be, e.g., an array including zeroes and occasional non-zero numbers. By calculating the first differential of edge distances and identifying non-zero points therein, the aspect ratio checker <NUM> may determine, based on the non-zero points of the first differential, where the aspect ratio of content regions is changing. Examples of such aspect ratio changes include the increase of the edge distance at time <NUM> corresponding to the positive first differential value <NUM>, the decrease of the edge distance at time <NUM> corresponding to the negative first differential value <NUM>, the increase of the edge distance at time <NUM> corresponding to the positive first differential value <NUM>, and the decrease of the edge distance at time <NUM> corresponding to the negative first differential value <NUM>. The aspect ratio checker <NUM> is configured to investigate such changes in aspect ratio identified based on the non-zero first differential values.

Returning to <FIG>, after determining the first differential <NUM> of the time series <NUM> and identifying where any of the (top, bottom, left, and/or right) components of the first differential <NUM> is non-zero (also referred to herein as points where the first differential of the time series is non-zero), the aspect ratio checker <NUM> further determines the aspect ratio of the content regions within the image frames that correspond to the non-zero points in the first differential <NUM>. It should be understood that each of the non-zero points in the first differential <NUM> occurs where the edge distance(s) of the content region in one image frame differs from the corresponding edge distance(s) of the content region in an immediately subsequent image frame. Such an immediately subsequent image frame is referred to herein as the "corresponding image frame," and the aspect ratio checker <NUM> determines the (changed) aspect ratio of a content region in the corresponding image frame. It should be understood that, calculating only the aspect ratios of content regions in images frames that correspond to non-zero points in the first differential <NUM> of the time series <NUM> is faster than calculating an aspect ratio of the content region in each image frame and comparing such an aspect ratio to an expected aspect ratio. In addition, the aspect ratio checker <NUM> is able to detect subtle shifts of only a few pixels based on non-zero first differential values, including shifts that occur on individual edge distance axes (e.g., where only the top or only the bottom of the content region shifts). Experience has shown that such subtle shifts may be missed with other techniques, such as comparing to determine whether the aspect ratio of a content region within each image frame is within a threshold of tolerance of an aspect ratio. For example, if the aspect ratio shifted to another aspect ratio and back again, but not quite the same as the original aspect ratio, then a comparison to determine whether the aspect ratio is within a threshold of tolerance of an aspect ratio may miss such a shift, whereas techniques disclosed herein based on non-zero points in the first differential of the time series would not.

As further shown, the aspect ratio checker <NUM> outputs a report <NUM>, which may indicate the determined aspect ratios and image frame ranges associated with those aspect ratios. In general, any suitable report <NUM> may be generated by the aspect ratio checker <NUM>. For example, the report <NUM> may include a user interface (UI), displayed via a display device, that lists time stamps of image frame ranges where particular aspect ratios occur, as determined based on the non-zero first differential points and aspect ratios determined for corresponding image frames. Each such image frame range may begin with an image frame for which an aspect ratios was determined, as a result of a corresponding non-zero first differential value, and end with a next image frame for which another aspect ratio was determined, as a result of another non-zero first differential value. The generated UI allows a user to see aspect ratio changes in the video <NUM>, which may be indicative of aspect ratio errors. Of course, other Uls, such as a UI that displays image frames whose aspect ratios differ from an expected aspect ratio, may be output for display in alternative configurations.

<FIG> illustrates a method <NUM> for detecting and reporting aspect ratio errors in a video, according to an aspect of this disclosure. As shown, the method <NUM> begins at operation <NUM>, where the aspect ratio checker <NUM> receives a video as input.

At operation <NUM>, the aspect ratio checker <NUM> splices the received video into component image frames. Any suitable splicing technique may be used to extract still image frames from the video, including splicing functions provided by publicly available video processing libraries such as the FFmpeg library.

At operation <NUM>, the aspect ratio checker <NUM> loops through and processes each of the image frames in turn. Although shown for illustrative purposes as a loop through each of the image frames, some configurations may utilize parallel processing instead. For example, the aspect ratio checker <NUM> may break a video into chunks of image frames and process each of the chunks in parallel.

During the processing of the image frames, the aspect ratio checker <NUM> determines whether one of the image frames is completely black at operation <NUM>. In one configuration, the aspect ratio checker <NUM> may compare (on, e.g., a row-by-row and pixel-by-pixel basis) color values of each pixel in the image frame to a value of the color black. In such a case, the aspect ratio checker <NUM> may determine that the image frame is completely black if the color values of pixels in the image frame all match the black color value up to a threshold of tolerance. As described, for a color model where color values can range from <NUM> to <NUM>, with <NUM> being true black, the threshold may be set to between, e.g., <NUM> and <NUM>.

If the image frame is completely black, then at operation <NUM>, the aspect ratio checker <NUM> skips the image frame, and the method <NUM> continues to operation <NUM>, where the aspect ratio checker <NUM> determines whether there are additional image frames to process. As described, completely black image frames do not include content regions whose aspect ratios should be checked for errors.

On the other hand, if the image frame is not all black, then at operation <NUM>, the aspect ratio checker <NUM> determines top, bottom, left, and right positions of a content region within the image frame. In one configuration, the top, bottom, left, and right positions may be determined as distances from top, bottom, left, and right edges, respectively, of the image frame. In such a case, the aspect ratio checker <NUM> may normalize (x, y) coordinates determined for the top, bottom, left, and right of the content region to be distances from the top, bottom, left, and right edges of the image frame, respectively, as discussed in greater detail below. However, the top, bottom, left, and right positions may also be stored differently in alternative configurations, e.g., as (x, y) coordinates within the image frame. <FIG> illustrates the determining of the top, bottom, left, and right positions of the content region in greater detail, according to an aspect of this disclosure. As shown, at operation <NUM>, the aspect ratio checker <NUM> determines and stores a top coordinate of the content region within the image frame, where a top black bar ends and the content region begins. In one configuration, the aspect ratio checker <NUM> starts from the top of the image frame and checks, on a row-by-row and pixel-by-pixel basis, whether values of pixels in a row all match the value for black up to the threshold of tolerance. Once again, the threshold may be set to between, e.g., <NUM> and <NUM> for a color model in which color values can range from <NUM> to <NUM> and <NUM> is true black. If all of the pixels in a row are within the threshold of black, then the aspect ratio checker <NUM> may consider such a row to be completely black and move on to the next row. On the other hand, if any non-black pixels are detected in a row, then the processing breaks and the aspect ratio checker <NUM> stores the coordinate (i.e., row count) of that row as the top coordinate.

At operation <NUM>, the aspect ratio checker <NUM> determines and stores a coordinate of a bottom of the content region within the image frame. Operation <NUM> is similar to operation <NUM>, except the aspect ratio checker <NUM> starts from the bottom of the image frame and checks, on a row-by-row and pixel-by-pixel basis, whether the values of pixels in a row all match a value for black up to the threshold of tolerance. The aspect ratio checker <NUM> skips such completely black rows and stores a coordinate (i.e., the row count from the bottom) of a first row the aspect ratio checker <NUM> reaches with at least one non-black pixel as the bottom of the content region.

At operation <NUM>, the aspect ratio checker <NUM> rotates the image frame by ninety degrees in order to determine positions of the left and right sides of the content region in the image frame. Then, at operations <NUM>-<NUM>, the aspect ratio checker <NUM> essentially repeats the operations <NUM>-<NUM> of determining and storing coordinates of a bottom and a top of the content region within the rotated image frame in order to obtain the left and right coordinates of the content region, respectively. As the image frame has been rotate by ninety degrees, the coordinates of the bottom and top of the content region in the rotated image frame that are determined at operations <NUM>-<NUM> are actually coordinates of the left and right edges of the image frame prior to the rotation. Subsequent to step <NUM>, the aspect ratio checker <NUM> has coordinates for each of the top, bottom, left, and right edges where black bars end and a content region begins.

At operation <NUM>, the aspect ratio checker <NUM> normalizes the determined top, bottom, left, and right coordinates so that the positions of the top, bottom, left, and right of the content region are expressed as distances from the top, bottom, left, and right edges of the image frame, rather than, e.g., (x,y) coordinates of the entire image frame. Example code for such a normalization is described above, and the normalization can make interpretation of a time series easier.

Returning to <FIG>, at operation <NUM>, the aspect ratio checker <NUM> determines if there are more image frames in the video to process, in which case the method <NUM> returns to operation <NUM>, where the aspect ratio checker <NUM> continues looping through and processing the (remaining) image frames. On the other hand, if there are no more image frames in the video to process, then the method <NUM> continues to operation <NUM>, where the aspect ratio checker <NUM> generates a time series using the determined top, bottom, left, and right positions of content regions within the image frames. In one configuration, such a time series may indicate, for each image frame number (of image frames that are not completely black), the determined distances from the image frame edges of the top, bottom, left, and right of the content region within the image frame. In another configuration, the top, bottom, left, and right edge distances may also be separated into distinct time series, with each of those time series being processed in a similar manner as a time series with the top, bottom, left, and right edge distances combined.

At operation <NUM>, the aspect ratio checker <NUM> determines a first differential of the time series generated at step <NUM>. As described, the first differential of the time series represents the rate of change of the top, bottom, left, and right positions of the content regions at each relevant point in time.

Then, at operation <NUM>, the aspect ratio checker <NUM> determines points where any of the (top, bottom, left, and/or right) components of the first differential of the time series has a non-zero value. The non-zero first differential values are indicative of changes to the aspect ratio of content regions within successive image frames.

At operation <NUM>, the aspect ratio checker <NUM> determines the aspect ratios of content regions in image frames corresponding to the determined points where the first differential is non-zero. That is, the aspect ratio checker <NUM> determines a proportional relationship between the width and height (in pixels) of the content regions within image frames corresponding to non-zero values in the first differential of the time series. In a configuration in which the top, bottom, left, and right positions of content regions are expressed as edge distances, the aspect ratio checker <NUM> may determine the height of a content region within an image frame as the total height of the image frame minus the sum of the distances that the content region is from the top and bottom edges of the image frame, i.e., height = total_height - (top + bottom). In addition, the aspect ratio checker <NUM> may determine the width of the same content region within the image frame as the total width of the image frame minus the sum of the distances that the content region is from the left and right edges of the image frame, i.e., width = total_width - (left + right).

At operation <NUM>, the aspect ratio checker <NUM> reports, via a user interface displayed on a display device, the aspect ratios determined at step <NUM> and image frame ranges where those aspect ratios occur, each of the ranges beginning with an image frame for which an aspect ratios was determined and ending with a next image frame for which another aspect ratio was determined. For example, the aspect ratio checker <NUM> may generate (and cause to be displayed) a user interface that lists the time stamps of image frames for each of the determined aspect ratios. Other Uls, such as a UI that displays image frames whose aspect ratios differ from an expected aspect ratio, may be generated by the aspect ratio checker <NUM> in alternative configurations.

<FIG> illustrates a system <NUM> in which an aspect of this disclosure may be implemented. As shown, the system <NUM> includes, without limitation, a central processing unit (CPU) <NUM>, a network interface <NUM> connecting the system <NUM> to a network <NUM>, an interconnect <NUM>, a memory <NUM>, and storage <NUM>. The system <NUM> also includes an I/O device interface <NUM> connecting I/O devices <NUM> (e.g., keyboard, display and mouse devices) to the system <NUM>.

The CPU <NUM> retrieves and executes programming instructions stored in the memory <NUM>. Similarly, the CPU <NUM> stores and retrieves application data residing in the memory <NUM>. The interconnect <NUM> facilitates transmission, such as of programming instructions and application data, between the CPU <NUM>, I/O device interface <NUM>, storage <NUM>, network interface <NUM>, and memory <NUM>. The CPU <NUM> is included to be representative of a single CPU, multiple CPUs, a single CPU having multiple processing cores, one or more graphics processing units (GPUs), a combination of such processors, and the like. And the memory <NUM> is generally included to be representative of a random access memory. The storage <NUM> may be a disk drive storage device. Although shown as a single unit, the storage <NUM> may be a combination of fixed and/or removable storage devices, such as magnetic disk drives, flash drives, removable memory cards or optical storage, network attached storage (NAS), or a storage area-network (SAN). As shown, the storage <NUM> includes a video <NUM> that may be processed using the aspect ratio checker application <NUM>. Further, the system <NUM> is included to be representative of a physical computing system as well as virtual machine instances hosted on a set of underlying physical computing systems. Further still, although shown as a single computing system, one of ordinary skill in the art will recognized that the components of the system <NUM> shown in <FIG> may be distributed across multiple computing systems connected by a data communications network.

As shown, the memory <NUM> includes an operating system <NUM> and the aspect ratio checker application <NUM>. The operating system <NUM> may be, e.g., Linux® or Windows®. The aspect ratio checker application <NUM> is configured to recognize aspect ratio errors in the image frames of videos and report the same. In one configuration, the aspect ratio checker application <NUM> may receive a video as input; splice the received video into component image frames; loop through and processes each of the image frames by skipping completely black image frames and determining top, bottom, left, and right positions of a content region within each image frame that is not completely black; generate a time series using the determined top, bottom, left, and right positions of content regions within the image frames; determine a first differential of the generated time series; determine points where the first differential of the time series is non-zero; determine aspect ratios of content regions of image frames corresponding to the non-zero points; and report, via a user interface displayed on a display device, the determined aspect ratios and image frame ranges where those aspect ratios occur, according to the method <NUM> discussed above with respect to <FIG>.

Advantageously, techniques are disclosed for automatically recognizing and reporting aspect ratio errors in image frames of videos. Aspect ratio errors that can be recognized include (<NUM>) sudden changes from one aspect ratio to another after black bars are removed from the image frames, and (<NUM>) fluctuations in such an aspect ratio between image frames, including fluctuations that are difficult to perceive by the naked eye. Experience has shown that techniques disclosed herein may be faster and detect relatively minor aspect ratio changes that other approaches, such as comparing the aspect ratio of a content region within each image frame of a video to an expected aspect ratio while permitting a threshold of tolerance, may miss.

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various aspects of the present disclosure. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order or out of order, depending upon the functionality involved.

Claim 1:
A computer-implemented method for recognizing aspect ratio changes that are indicative of aspect ratio errors in a video (<NUM>) including a plurality of image frames (<NUM>), the method comprising:
determining (<NUM>), for each image frame (<NUM>) of the plurality of image frames (<NUM>), positions of a top, a bottom, a left, and a right of a content region (<NUM>) in the image frame (<NUM>), wherein the content region (<NUM>) is the image frame (<NUM>) exclusive of any black bars therein that have color values within a threshold from a color value for black;
generating (<NUM>) at least one time series (<NUM>) from the determined positions; and
determining (<NUM>, <NUM>, <NUM>) aspect ratio changes based on calculating only the aspect ratios of content regions in image frames of the plurality of image frames (<NUM>) corresponding to non-zero points in at least one first differential (<NUM>) of the at least one time series (<NUM>).