Playfield detection and shot classification in sports video

A method of classifying the shot type of a video frame, comprising loading a frame, dividing the frame into field pixels and non-field pixels based on a first playfield detection criteria, determining an initial shot type classification using the number of the field pixels and the number of the non-field pixels, partitioning the frame into one or more regions based on the initial classification, determining the status of each of the one or more regions based upon the number of the field pixels and the non-field pixels located within each the region, and determining a shot type classification for the frame based upon the status of each the region.

BACKGROUND

1. Technical Field

The present disclosure relates to the field of digital video analysis and encoding, particularly a method of detecting playfields and classifying shots in sports video.

Watching sports video is a popular pastime for many people. Digital transmissions of sports video can be viewed on televisions directly or through set-top boxes, or on other devices such as personal computers, tablet computers, smartphones, mobile devices, gaming consoles, and/or other equipment. Digital recordings of sports video can be viewed on the same devices and viewing the digital recordings can begin at the start of a recorded event or midway through the event.

Automatic parsing of sports video based on visual and/or cinematographic cues can be used to identify segments of potential interest to a viewer, and/or points at which video on demand playback can begin. Visual cues, such as long shots, medium shots and close up shots, can be used to identify segments of the video where on-field events are occurring, or to distinguish on-field events from close up views of players, referees, balls, logos, or other items. Long shots frequently provide coverage of large areas of a playing surface, such as a playing field or court, and frequently identify periods of time during which activity on the field is at a maximum. Extended periods of play can comprise a sequence of long shots followed by medium and/or close up shots which signify the end of a play or highlight the contributions of key players. Detection of long shots can also aid in automatically identifying highlights from the video, and/or automatically summarizing video.

Some methods of classifying shots have been developed. For example, some existing methods learn and adjust to color variations of the playfield, but do not detect shot types based on color histograms of selected regions of frames or accumulate the histograms by determined shot types. In other existing methods, color histograms are accumulated over a random selection of frames, not a selection of frames determined by the shot type. Still other methods use a Gaussian mixture model to classify shots, but requires training time to determine peaks of histograms before shot classification can begin, which can be difficult if non-sports video is interspersed with the sports video, such as commercials or pregame analysis.

SUMMARY

What is needed is a method of shot identification that can classify the shot type of a single frame of video without needing training time based on playfield detection criteria such as HSV color ranges, and can adapt the playfield detection criteria over time as additional frames are reviewed.

In one embodiment, a method of classifying the shot type of a video frame is provided, the method comprising loading a frame, dividing the frame into field pixels and non-field pixels based on a first playfield detection criteria, determining an initial shot type classification using the number of the field pixels and the number of the non-field pixels, partitioning the frame into one or more regions based on the initial classification, determining the status of each of the one or more regions based upon the number of the field pixels and the non-field pixels located within each the region, and determining a shot type classification for the frame based upon the status of each the region.

DETAILED DESCRIPTION

FIG. 1depicts an exemplary frame100of a long shot in sports video. A video frame100can comprise a plurality of pixels. By way of a non-limiting example, in some 1080p high definition videos, each frame100can comprise 1920 pixels by 1080 pixels. Long shots can be shots of one or more subjects that are taken at a distance and/or have a wide field of view. In sports video, long shots generally show a portion of a playfield102, as shown inFIG. 1. In some sports, the playfield102can be a court, such as in basketball or tennis. In other sports, the playfield102can be a field, such as in football, baseball, and soccer. In still other sports, the playfield102can be an ice rink, such as in ice hockey, or other type of playfield. Playfields102can be made of grass, turf, wood, ice, cement, clay, sand, any other type of surface. Regardless of the specific sport, most playfields102have large areas of uniform or near uniform color. By way of non-limiting examples, in football most fields are made of grass or turf that is predominantly green, while in basketball many courts are made of hardwood that predominantly has natural tan coloring. When long shots and medium shots can be identified in sports video, this information can be used to influence the operations of video processing equipment that includes video encoders and/or transcoders. For example, in some embodiments a video encoder can modify its operating parameters such as quantization step size, coding unit size, transform unit size and macro-block modes based on the presence of a long shot being signaled. The uniform or near uniform color of playfields102can enable identification of long shots and/or medium shots in video frames, such that the video frames100can be encoded with bits allocated such that visually important parts of the frame100, for instance the playfield102and players, are encoded at higher qualities using the available bit budget.

As can be seen fromFIG. 1, in some shots the playfield102can be at least partially obscured by offensive and/or defensive players104, referees, balls, lines106, logos, on-screen displays108, or other items. In some shots other elements or backgrounds can be visible, such as fans, stands, sidelines, portions of a stadium, or other items. Even though other items can be visible in the frame100, in most long and/or medium shots the uniform or near uniform color of the playfield102can still fill a significant portion of the frame100, as shown inFIG. 1.

FIG. 2depicts a method for determining that a video frame100depicts a playfield102and for classifying the frame100as a long shot and/or medium shot. At step200, a system can instantiate playfield detection criteria110in memory, and set the playfield detection criteria110to predetermined initial playfield detection criteria110. The system can be a video processor, encoder, server, computer, or any other piece of equipment that can receive and analyze video frames100. By way of a non-limiting example, the system can be an embedded system comprising one or more processors and one or more digital signal processors.

In some embodiments, the playfield detection criteria110can be color ranges defined in color models such as RGB, HSV, or any other desired color model. RGB models can describe the color of a pixel using the pixel's level of red, green, and blue. HSV models break down the color of a pixel into separate hue (H), saturation (S), and value (V) components. The hue component can describe the pixel's hue or tint. The saturation component can describe the pixel's saturation or amount of gray. The value component can describe the pixel's brightness or color intensity.

In some embodiments the system can set the initial playfield detection criteria110to be one or more color ranges that describe colors that are expected to appear within the playfield102of the particular sporting event and/or venue being shown in the video. By way of a non-limiting example, if the video is expected to depict a basketball game being played on a hardwood court playfield102that has natural tan coloring, the system at step200can set the initial playfield detection criteria110to a predetermined color range describing the colors generally expected to appear in hardwood basketball courts having natural tan coloring. By way of another non-limiting example, the initial playfield detection criteria110can also include color ranges generally expected to appear in lights that are reflected in a basketball court, such as a range of purple colors that have a high concentration of value components.

In some embodiments, the system can set the initial playfield detection criteria110to initial HSV ranges of the colors that are expected to appear within the playfield102of the particular sporting event and/or venue being shown in the video. By way of a non-limiting example,FIGS. 3A-3Cdepict an example of predetermined initial HSV ranges for naturally colored hardwood basketball courts, withFIG. 3Adepicting an initial hue range302out of the possible hue component values304,FIG. 3Bdepicting an initial saturation range306out of the possible saturation component values308, andFIG. 3Cdepicting an initial value range310out of the possible value component values312. As can be seen fromFIGS. 3A-3Cin some embodiments the hue range302can be narrower than the saturation range306and value range310because playfield pixels can generally have more variation in saturation and value than in hue.

As will be discussed below, in some embodiments the initial playfield detection criteria110can be updated and/or refined as additional frames100are considered by the system during the video, such that the playfield detection criteria110are adapted over time to the colors of the specific playfield102of the particular sporting event and/or venue being shown in the video. By way of a non-limiting example, during analysis of a video the system can narrow one or more of the initial HSV ranges302,306, or310to fit the colors determined to be in the playfield102over time as additional frames100are considered.

Returning toFIG. 2, at step202the system can receive and/or load a video frame100comprising a plurality of pixels. The video frame100can be received and/or loaded from a source, such as a camera, media server, receiver, or any other piece of equipment. The frame100received and/or loaded at step202can be a frame100from any point in the video. By way of a non-limiting example, the system at step202can receive and/or load the first frame100of the video, proceed with other steps shown inFIG. 2to determine whether a playfield100is depicted and/or classify the frame100as being a long shot or medium shot, and then return to step202to receive and/or load the next frame100of the video and repeat the process for the next frame100.

At step204, the system can divide the frame100into field pixels402and non-field pixels404based on the playfield detection criteria110. Those pixels that meet the playfield detection criteria110can be classified as field pixels402, and those pixels that do not meet the playfield detection criteria110can be classified as non-field pixels404. By way of a non-limiting example, when the playfield detection criteria110includes a set of HSV ranges, the pixels of the frame100that have hue, saturation, and value components that are all within the hue range302, the saturation range306, and the value range310can be determined to be field pixels402, while the pixels of the frame100that have at least one hue, saturation, or value component outside of the hue range302, the saturation range306, and the value range310can be determined to be non-field pixels404.

In some embodiments, at step204the system can divide the frame100into field pixels402and non-field pixels404by generating a binary mask406of the frame100. The binary mask406can comprise mask pixels representing the pixels classified as field pixels402, and non-mask pixels representing the pixels classified as non-field pixels404. By way of a non-limiting example,FIG. 4Adepicts an exemplary frame100, andFIG. 4Bdepicts an exemplary binary mask406comprising white mask pixels representing the field pixels402determined fromFIG. 4Aand black non-mask pixels representing the non-field pixels404determined fromFIG. 4A.

In some embodiments, when the playfield detection criteria110contains more than one set of HSV color ranges, each set of HSV ranges can be considered separately to sort pixels into mask pixels and non-mask pixels to generate a binary mask406for each set of HSV ranges, and each binary mask406can be combined into a final combined binary mask406. Further, in some embodiments, a set of HSV ranges can be applied to a region of the frame100rather than the whole frame100to generate a binary mask406.

When two or more sets of HSV ranges are in the playfield detection criteria110, the first set of HSV ranges can be considered a first playfield detection criteria110aand the second set of HSV ranges can be considered a second playfield detection criteria110b. The first set of HSV ranges can be different than the second set of HSV ranges, and therefore the first playfield detection criteria110ais different from the second playfield detection criteria110b. Field pixels402for a frame100can be obtained by applying the first playfield detection criteria110atoafirst region and the second playfield detection criteria110btoasecond region. The field pixels402for the second region of the frame100can be incorporated into the field pixels402obtained from the first playfield detection criteria110a, such that the field pixels402for the second region of the frame100is incorporated into the field pixels402obtained from the first playfield detection criteria110ausing a logical combination of field pixels402from the first and second regions. The resulting final binary mask406can be the combination of binary masks406for each set of HSV ranges using logical AND, OR, and/or XOR operations. By way of a non-limiting example, in some embodiments the playfield detection criteria110can include a set of HSV color ranges for a basketball court and a second set of HSV color ranges for gloss surface reflections within the basketball court, such as reflections of lighting fixtures, billboards, and/or other elements within the venue. The mask pixels and non-mask pixels of the final combined binary mask406can be considered to be the field pixels402and non-field pixels404, respectively.

Returning toFIG. 2, at step206the system can determine an initial classification of the shot type of the frame100. The shot type can be a long shot, a medium shot, or a close-up shot. In some embodiments, the system can determine the initial shot type classification using the number of the field pixels402and the number of the non-field pixels404. By way of a non-limiting example, in some embodiments the system can determine the initial shot type classification by calculating a total field pixel ratio of the entire frame100from the number of field pixels402and the number of the non-field pixels404. The system can obtain the total field pixel ratio of the entire frame100by dividing the number of field pixels402by the total number of pixels in the frame100. In some embodiments, the total field pixel ratio can be compared against one or more predetermined total threshold ratios for one or more shot types to obtain the initial classification of the shot type. By way of a non-limiting example, in some embodiments, the predetermined total threshold ratio for a long shot can be set at 20%, such that if the total field pixel ratio is at or above 20%, the frame100can be initially classified as a long shot at step206.

At step208, the system can determine if the frame100was initially determined to be a long shot or medium shot at step206. If the frame100was determined to be a long or medium shot, the system can move to step210. If the frame100was not determined to be a long or medium shot, the system can move to step212to inform the encoder that the frame100is not a long or medium shot, and/or to inform the encoder to encode the frame100normally without considering the frame100to be a long or medium shot. If the frame100was not determined to be a long or medium shot, the system can also return to step202to receive and/or load the next frame100.

At step210, the system can determine a final classification of the shot type of the frame. In some embodiments, the system can refine and/or verify the initial shot type classification determined during step206to obtain the final shot type classification. To determine the final shot type classification, the system can partition the frame100into one or more detection regions500based on said initial classification, and determine one or more regional field ratios of one or more the detection regions500of the frame100. In some embodiments, the detection regions500can be geometrically constrained regions, such as rectangular regions, square regions, or any other geometric region. In alternate embodiments, the detection regions500can have curved edges, be oval, round, be polygonal, or have any other desired shape. By way of a non-limiting example,FIG. 5Adepicts an embodiment in which the total frame100is divided vertically into two regions: the top quarter, and the bottom three quarters, in which the bottom three quarters can be a first detection region500a. As another non-limiting example,FIG. 5Bdepicts an embodiment in which the total frame100is divided horizontally into three regions: the left quarter, the center half, and the right quarter, in which the center half can be a second detection region500b.

The system can determine the status of each of one or more detection regions500based upon the number of field pixels402and non-field pixels404located within each detection region50. In some embodiments, the regional field ratio of each detection region500can be obtained by dividing the number of field pixels402within the detection region500by the total number of pixels within the detection region500. In alternate embodiments, the regional ratio of each detection region500can be obtained by dividing the number of field pixel402within one sub-region by the number of field pixel402within another sub-region. By way of a non-limiting example, in some embodiments, the regional field ratio can be determined by dividing the number of field pixels402within the top quarter of the frame100by the number of field pixels402within the bottom three quarters of the frame100. The regional field ratio for each detection region500can be compared against one or more predetermined regional threshold ratios for one or more shot types to confirm or update the initial classification of the shot type and determine the final shot type classification at step210. By way of a non-limiting example, in some embodiments the predetermined regional threshold ratio for the first detection region500ashown inFIG. 5Afor a long shot can be set at 65%, such that if the regional field pixel ratio is at or above 65% within the first detection region500a, the final classification of shot type of the frame100can be determined to be a long shot at step210.

In some embodiments, the system can determining the final shot type classification for the frame100based upon the status of each detection region500. By way of a non-limiting example, in some embodiments the system can determine the final shot type classification by comparing the regional field ratios of more than one detection region500against each detection region's predetermined regional threshold ratio. In other embodiments, the system can determine the final shot type classification by comparing the regional field ratios of a single detection region500against that detection region's predetermined regional threshold ratio.

In alternate embodiments, if the total field pixel ratio determined during step206was above a histogram threshold percentage, a hue histogram of the frame100can be used to verify the initial shot type classification. By way of a non-limiting example, the histogram threshold percentage can be set at 50%. The frame's hue histogram can be partitioned into two regions: a first range within the hue range of the playfield detection criteria110, and a second range outside of the hue range or the playfield detection criteria110. In some embodiments, if the standard deviation of the second range is larger than the standard deviation of the first range, the frame100can be classified as a medium shot regardless of the regional field ratios. In alternate embodiments, a standard deviation ratio can be calculated by dividing the standard deviation of the second range by the standard deviation of the first range. The standard deviation ratio can be compared to a predetermined standard deviation ratio threshold to determine a shot type classification. For example, if the standard deviation ratio is larger than one, the frame100can be classified as a medium shot regardless of the regional field ratios.

FIGS. 4A-4B,6A-6B, andFIGS. 7A-7Bdepict non-limiting examples of video frames100and binary masks that can be generated from the video frames100based on the playfield detection criteria110.

FIG. 4Adepicts a frame100that depicts a wide shot of a basketball game broadcast that shows the playing surface102. The frame100ofFIG. 4Acan be determined to be a long shot through the steps of202-210.FIG. 4Bdepicts a binary mask of field pixels402and non-field pixels404that can be generated at step204from the frame100shown inFIG. 4A. At step206, the system can determine that the total field pixel ratio is high enough to initially qualify the frame100as a long shot. At step210, the system can confirm the initial classification and determine the final shot type classification as a long shot by finding that the regional field pixel ratios in both the first detection region500aof the bottom three quarters of the frame100and the second detection region500bof the center half of the frame100are high enough to qualify the frame100as a long shot.

FIG. 6Adepicts a frame100that depicts on screen graphics during a basketball game broadcast, but does not show the playing surface102. The frame100ofFIG. 6Acan be determined to not be a long shot through the steps of202-210.FIG. 6Bdepicts a binary mask of field pixels402and non-field pixels404that can be generated at step204from the frame100shown inFIG. 6A. At step206, the system can determine that the total field pixel ratio is high enough to initially qualify the frame100as a long shot. However, the system can find at step210that the initial classification of the frame100as a long shot was incorrect, because the regional field pixel ratios in the first detection region500aof the bottom three quarters of the frame100was not high enough to qualify the frame100as a long shot. Alternatively, the system can find at step210that the initial classification of the frame100as a long shot was incorrect, because the field pixels402were more densely distributed in the top quarter of the frame100than in the bottom three quarters of the frame100, such that the regional pixel ratio determined by dividing the number of field pixels402it the top quarter of the frame by the number of field pixels402in the bottom three quarters of the frame was higher than a predetermined regional threshold ratio.

Similarly,FIG. 7Adepicts a frame100that depicts announcers during a basketball game broadcast, but does not show the playing surface102. The frame100ofFIG. 7Acan be determined to not be a long shot through the steps of202-210.FIG. 7Bdepicts a binary mask of field pixels402and non-field pixels404that can be generated at step204from the frame100shown inFIG. 7A. At step206, the system can determine that the total field pixel ratio is high enough to initially qualify the frame100as a long shot. However, the system can find at step210that the initial classification of the frame100as a long shot was incorrect, because the regional field pixel ratios in the first detection region500aof the bottom three quarters of the frame100was not high enough to qualify the frame100as a long shot. Alternatively, the system can find at step210that the initial classification of the frame100as a long shot was incorrect, because the field pixels402were more densely distributed in the top quarter of the frame100than in the bottom three quarters of the frame100, such that the regional pixel ratio determined by dividing the number of field pixels402in the top quarter of the frame by the number of field pixels402in the bottom three quarters of the frame was higher than a predetermined regional threshold ratio.

Returning toFIG. 2, at step212the system can inform an encoder of the shot type of the frame100. If the final shot type classification of the frame100was a long shot or medium shot, the system can inform the encoder that the frame100is a long shot or medium shot. If the initial and/or final shot type classification of the frame100was not a long shot or medium shot, the system can inform the encoder that the frame100is not a long or medium shot, and/or inform the encoder to encode the frame100normally without considering the frame100to be a long or medium shot. In some embodiments, the encoder can use the final shot type classification to encode the frame100. In some embodiments, the system can additionally inform the encoder about which pixels of the frame100were classified as field pixels402and/or non-field pixels404. By way of a non-limiting example, the system can transmit the frame's binary mask406to the encoder.

At step214, the system can determine whether the final shot type classification determined at step210was a long shot or medium shot. If the final shot type classification was not a long shot or medium shot, the system can return to step202to receive and/or load the next frame100. However, if the final shot type classification was a long shot or medium shot, the system can move to steps216-220to update the playfield detection criteria110stored in memory based on the color characteristics of the frame100.

At step216the system can generate one or more local histograms800of the colors of one or more selected regions802of the frame100. The selected regions802can be predetermined areas of the frame100likely to show at least a portion of the playfield102. In some embodiments, the selected regions802can be determined based at least in part on the final classification of the shot type and/or the sport being shown in the video. By way of a non-limiting example, for frames100of a basketball game video that were determined to be long shots, as shown inFIG. 8Athe selected region802can be the lowest three quarters of the frame100because in basketball video the upper quarter of the frames100are likely to be non-court area. In alternate embodiments, the selected region802can be the same as one of the detection regions500used in step210.

In embodiments in which the playfield detection criteria110are HSV ranges, local histograms800of each of the hue, saturation, and value components of each of the pixels in the selected region802can be generated. By way of a non-limiting example,FIG. 8Adepicts a video frame100, andFIG. 8Bdepicts a binary mask406that can be used to determine that the frame shown inFIG. 8Ais a long shot.FIGS. 8C-8Erespectively depict a local hue histogram800a, a local saturation histogram800b, and a local value histogram800cof the pixels within a selected region802within the frame100shown inFIG. 8A.

At step218, the local histograms800generated during step214for a single frame100of a particular shot type can be integrated into accumulated histograms900for the shot type. By way of a non-limiting example, a local histogram800of a frame100determined to be a long shot can be integrated into an accumulated histogram900of all frames100determined to be long shots. If no previous frames100for the shot type have been analyzed, the local histograms800of the first frame100of the shot type can be used as the accumulated histogram900for that shot type. By way of a non-limiting example, in alternate embodiments, if no previous frames100for the shot type have been analyzed, uniform distribution within the color range of the play field detection criteria110can be used as the accumulated histogram900.

At step220, the system can update the playfield detection criteria110in memory based on the accumulated histograms900for each shot type. One or more peaks904of the accumulated histograms900can be determined, and the playfield detection criteria110can be fit to the peaks904. In some embodiments, the system can perform low-pass filtering on the values of the accumulated histogram900prior to finding peaks904. By way of a non-limiting example, in equation form, Hnew(i)=0.25H(i−1)+0.5H(i)+0.25H(i+1), where H is the accumulated histogram900and Hnew is the filtered accumulated histogram900. By way of a non-limiting example, if the playfield detection criteria110includes HSV color ranges that are too broad for the peaks904of the accumulated local histogram900, the HSV color ranges can be narrowed to fit the peaks904, and be and saved as updated playfield detection criteria110. The updated playfield detection criteria110can be saved to memory, and be used by the system at step204when the next frame100is analyzed and divided into field pixels402and non-field pixels404.

By way of a non-limiting example,FIGS. 9A-9Crespectively depict an accumulated hue histogram900a, an accumulated saturation histogram900b, and an accumulated value histogram800cfor long shots that incorporates data from the local histograms800shown inFIGS. 8C-8E.FIG. 9Adepicts an accumulated histogram900aof the hue components of pixels in the selected area802,FIG. 9Bdepicts an accumulated histogram900bof the saturation components of pixels in the selected area802, andFIG. 9Cdepicts an accumulated histogram900cof the value components of pixels in the selected area802. As can be seen fromFIG. 9A, the initial hue range302awas too wide for the peak904of the accumulated histogram900a, so the hue range can be narrowed to the updated hue range302b. Similarly, as can be seen fromFIGS. 9B and 9C, the initial saturation range306aand the initial value range310awas too wide for the peaks904of the accumulated histograms900band900c, so the saturation and/or value ranges can be narrowed to the updated saturation range306band/or the updated value range310b. The updated hue range302b, the updated saturation range306b, and the updated value range310bcan be used as the updated playfield detection criteria110. As discussed above, in some embodiments the pixels depicting a playfield102generally have more variation in saturation and/or value than in hue, so the updated hue range302bcan be narrower than the updated saturation range306band/or the updated value range310b. In some embodiments, the updated hue range302bcan provide cleaner playfield segmentation when the binary mask404is generated in step204.

In some embodiments, dominant peaks can be found in the accumulated histograms900within the HSV ranges, and the dominant peaks can be used to narrow the HSV ranges, for example from the initial playfield detection criteria110. In alternate embodiments, the total distribution of the accumulated histograms900within the playfield detection criteria110color ranges can be obtained and used to narrow the color ranges. By way of a non-limiting example, for the accumulated hue histogram900ashown inFIG. 9A, the total distribution (S) within the initial hue range302acan be obtained by summing the histogram values (H) between the low end (H_lo) of the hue range302aand the high end (H_hi) of the hue range302a. In equation form, the total distribution S=sum(H[H_lo:H_hi]). The initial hue range302acan be updated to the updated hue range302b, such that the updated hue range302bspans between a new low end (new_H_lo) and a new high end (new_H_hi) in which the sum of the histogram values between the new low end and the new high end is less than (S*T), where T is a predetermined threshold. In some embodiments, the predetermined threshold (T) can be a value less than 1.0. In equation form, the initial hue range302acan be narrowed to an updated hue range302bin which |sum(H[new_H_lo:new_H_hi])−S*T|<epsilon, where T<1.0 and epsilon is a small positive float number, such as 0.01. By way of a non-limiting example, the predetermined threshold (T) can be 0.95, such that after narrowing the range down to the updated hue range302b, 95% of the accumulated distribution within the initial hue range302ais preserved. Playfield detection criteria110can be updated at step220to the new updated hue range302bbetween the new low end (new_H_lo) and the new high end (new_H_hi).

In some embodiments, when the video frame100depicts a basketball game, the main peak of the accumulated value (V) histogram900ccan be identified at the bright end with the value range310extending between a first boundary (Vt) and the largest value (255). In these embodiments, the updated value range310bcan be determined by changing the first boundary of the initial value range310ato an updated first boundary (Vt) in the updated value range310b, while keeping the second boundary fixed at the largest value (255), such that the value range310is narrowed only from one end. In some embodiments, the updated first boundary (Vt) can be found by using Otsu's threshold method on the accumulated value histogram900cto identify a first threshold value (v1). Otsu's threshold method can be used again on the accumulated value histogram900cwithin the range extending from the first threshold value (v1) and the largest value (255) to find a second threshold value (v2). The second threshold value (v2) can be used as the updated first boundary (Vt) for the updated value range310b, such that the range between the updated first boundary (Vt) and the largest value (255) is tight around the largest peak904in the accumulated value histogram900cat the bright end.

In alternate embodiments, the updated first boundary (Vt) for the updated value range310bof the accumulated value histogram900ccan be found within a range extending between a low value (S1) and a high value (S2), in which the low value and high value are obtained by the following equations: sum(V[0:S1])>=T1 and sum(V[0:S1−1])<T1, and sum(V[0:S2])>=T2 and sum(V[0:52−1])<T2. In some embodiments, T1 can be 0.3 and T2 can be 0.6. The updated first boundary (Vt) can be the minimum point between the low value (S1) and the high value (S2) of the accumulated value histogram900c. The initial value range310ain the playfield detection criteria110can be updated to an updated value range310bbetween the first boundary (Vt) and the highest possible value (255).

FIGS. 10-12depict further exemplary embodiments of frames100that can be determined to be long shots using the steps ofFIG. 2. These exemplary frames100can each have enough pixels determined to be field pixels402based on initial or updated playfield detection criteria110to initially qualify as a long shot, and can also have enough field pixels402in the detection regions500to have the system's final shot type classification be a long shot.

FIGS. 13-14depict exemplary embodiments of frames100that can be determined to be medium shots using the steps ofFIG. 2. These exemplary frames100can each have enough pixels determined to be field pixels402based on initial or updated playfield detection criteria110to initially qualify as a medium shot, pass the standard deviation ratio test, or also have enough field pixels402in the detection regions500to have the system's final shot type classification be a medium shot.

FIG. 15depicts an exemplary embodiment of a frame100that can be determined to not be a long or medium shot using the steps ofFIG. 2. Although this exemplary frame100can have some pixels found to be field pixels402based on initial or updated playfield detection criteria110, the system can find that the frame100does not have a total field pixel ratio high enough to classify the frame100as a long or medium shot.

As discussed above, in some embodiments the playfield102can be a basketball court. Basketball courts have bounded areas called keys1700underneath each basket, in which the three seconds rule is enforced. In some basketball courts, the keys1700have a different color than the remainder of the court. By way of a non-limiting example, in some basketball courts the keys1700can be painted green while the rest of the court is a natural hardwood color or stain.

In some embodiments in which the playfield102is expected to be a basketball court having a differently colored key1700than the rest of the court, the method ofFIG. 2can move to step1600after determining at step210that the shot is a long shot to further determine and verify the color ranges of the keys1700. In some embodiments, the determining and verification of the color ranges of the keys1700can be completed over a predetermined number of different frames100of long shots, and that depict specific camera angles such as a left camera view or right camera view. In some embodiments, after the color ranges of the keys1700have been determined and verified over the predetermined number of different frames100, the color ranges of the keys1700can be added to the playfield detection criteria110in step220, such that in subsequent frames100the presence of the keys1700can be identified within the frame100. By way of a non-limiting example, the initial playfield detection criteria110can be the color ranges for the main color of the court, without color information about the color of the keys1700. Once the presence of keys1700has been detected in the video and the color range of the keys1700has been determined and verified over the predetermined number of different frames, the initial playfield detection criteria110can be updated to include the color information of the keys1700.

FIG. 16depicts a method for estimating the color range of the keys1700in long shots. In some embodiments, the method ofFIG. 16can operate in parallel with steps216-220ofFIG. 2. If the system has determined at step210that the final shot type classification of a frame100is a long shot, the system can move to step1600to pass the frame100to a key estimation system. In some embodiments, the key estimation system can be component of the system. In other embodiments, the key estimation system can be a separate system. The key estimation system can activate step1602to determine whether the frame100depicts a left or right camera view. A right view can depict at least a portion of the key1700on the right side of the frame100, while a left view can depict at least a portion of the key1700on the left side of the frame100. By way of a non-limiting example,FIG. 17depicts a long shot frame100with a left side view showing a key1700. If the system determines that the frame100depicts a left or right view, the system can move to step1604. If the system determines that the frame100does not depict a left or right view, such as a view of the center of the court that does not show a portion of the key1700, the system can move to step202to load the next frame.

In some embodiments, the system at step1602can use the binary mask406generated during step204to determine the camera view based on the distribution of field pixels402in the total frame100. If the color characteristics of the keys1700have not yet been determined, the system can find that pixels showing the keys1700have color components outside of the playfield detection criteria110, and therefore classify the pixels of the keys1700as non-field pixels404. By way of a non-limiting example,FIG. 18depicts a binary mask406of the frame ofFIG. 17, in which the pixels of the key1700were found to be non-field pixels404, and the field pixels402were found to be the pixels with colors similar to the non-key areas of the playfield102. The system can look at the distribution of field pixels402and non-field pixels404in the binary mask406to determine which areas of the frame100have more field pixels402, and compare that distribution to expected models for left and right camera views. By way of a non-limiting example, in a left camera view the key1700can be expected to be on the left side of the frame100and the non-key areas of the court can be expected to make up a large portion of the right side of the frame100. The system can thus determine that a frame100depicts a left camera view when the left half of the frame100has fewer field pixels402than the left half of the frame100. In some embodiments, the distribution of field pixels402and non-field pixels404can be determined in a detection region, such as the vertically center portion of the frame100.

At step1604, the system can determine if the frame100was found to be a left camera view or a right camera view during step1602. If the frame100was not found to be a left camera view or a right camera view, the system can return to step202to load the next frame. While in this situation the system moves from step1604to step202, the system can have been concurrently and/or independently performing the steps of214-220before the next frame is loaded at step202. If the frame100was found to be a left camera view or a right camera view, the system can move to step1606.

At step1606, the system can generate a seed mask1900showing seed pixels1902and non-seed pixels1904. In some embodiments in which the frame100was determined to be a long shot with a left or right camera view, the system can generate the seed mask1900by first determining a maximum area contour for the field pixels402in the binary mask406. The system can then generate an initial seed mask, which is defined as the pixels from the convex hull of the maximum area contour. The system can then exclude the field pixels402from the initial seed mask to obtain the final seed mask1900. In some embodiments, the final seed mask1900can be generated by the binary XOR operation between the initial seed pixels and the field pixels402in the binary mask406. The seed pixels1902of the seed mask1900can represent pixels forming portions of the key1700, as well as other background pixels, such as text, fans, logos, or other elements. By way of a non-limiting example,FIG. 19depicts the seed mask1900of the frame100shown inFIG. 17, with the seed pixels1902shown in white and the non-seed pixels1904shown in black. In some embodiments, the seed mask1900can be generated from the half of the frame100that shows the key1700, which can be determined by the camera view. By way of a non-limiting example, the seed mask1900shown inFIG. 19was generated from the left half of the frame100ofFIG. 17because the frame100ofFIG. 17was determined to be a left camera view.

At step1608, the system can generate one or more local histograms800of the color components of the pixels in the seed mask1900. As with the local histograms800of the field pixels402discussed above with respect to step216, the local histograms800of the seed mask1900can be one or more histograms800of the colors of one or more selected regions802of the frame. In some embodiments, the selected regions802of the frame used in step1608can be the portions of the frame100defined by the seed pixels1902shown in the seed mask1900. By way of a non-limiting example,FIG. 20depicts a local histogram800of the hue components of the field pixels1902.

At step1610, the local histograms800of the seed masks1900generated during step1608for a single frame100can be integrated into accumulated histograms900for either the left or right camera views. The system can maintain accumulated histograms900for long shots with either right or left camera views. By way of a non-limiting example,FIG. 21depicts an accumulated histogram900of the hue values of a plurality of seed masks1900over multiple frames100of either a left or right camera view, including the local histogram800shown inFIG. 20. In some embodiments, the system can use a temporal recursive filter to integrate the data in the local histogram800of the key mask1900of the latest frame100into the accumulated histogram900for the camera view. For key color estimation, the accumulated histogram900can be accumulated over a predetermined number of frames for the accumulated histogram's camera view.

At step1612, the system can determine whether the accumulated histograms900have incorporated data from the predetermined number of frames100. In some embodiments, the system can check whether the accumulated histograms900have been generated from at least a predetermined number of local histograms800. By way of a non-limiting example, the predetermined number can be 180, such that the system can determine whether the local histograms800of at least 180 frames100have been accumulated into the accumulated histograms900for either left or right camera view.

If the accumulated histograms900have not yet incorporated data from the predetermined number of frames100, the key color estimation system can wait for more frames100, and return to step202to load the next frame. While in this situation the system moves from step1612to step202, the system can have been concurrently and/or independently performing the steps of214-220before the next frame is loaded at step202. If the accumulated histograms900have incorporated data from the predetermined number of frames100, the system can move to step1614.

At step1614, the system can determine candidate color ranges for the keys1700from the accumulated histograms900. The histograms900can have one or more peaks904that can be used to determine the candidate color ranges. In some embodiments, the system can perform low-pass filtering on the values of accumulated histogram900prior to finding peaks904. By way of a non-limiting example, in equation form, Hnew(i)=0.25H(i−1)+0.5H(i)+0.25H(i+1), where H is the accumulated histogram900and Hnew is the filtered accumulated histogram900. In some embodiments, the system can determine the color ranges for the keys1700using the method ofFIG. 22. The color ranges can be defined in a color format, such as HSV ranges.

At step2200, the system can search an accumulated histogram900to find a peak904. The maximum histogram value (V) of the histogram900and its index level (i) can be saved into memory.

At step2202, the system can determine a range between a low index level (10) and a high index level (hi), such that the index level (i) of the histogram's maximum value (V) is between the low index level (lo) and the high index level (hi). In equation form, the range can be: lo<=i<=hi.

The low index level (lo) and the high index level (hi) can be determined such that the histogram's value at the low index level (lo) is less than the histogram's maximum value (V) multiplied by an amplitude threshold (T), and also that the histogram's value at the high index level (hi) is less than the histogram's maximum value (V) multiplied by the amplitude threshold (T). In equation form, the low index level (lo) and the high index level (hi) can be determined such that: H(lo)<V*T and H(lo+1)>=V*T, and H(hi)<V*T and H(hi−1)>=V*T. The amplitude threshold (T) can be set at any desired value. By way of a non-limiting example, T can be set to 0.1, such that the values of the histogram at the low index level (lo) and the high index level (hi) are 10% of the histogram's maximum value (V).

At step2204, the system can determine the total distribution (S) of the range, by summing the values of the histogram800between the low index level (lo) and the high index level (hi). In equation form, the total distribution can be determined such that: S=sum(H[lo:hi]).

At step2206, the index (i), maximum value (V), low index level (lo), high index level (hi), and total distribution (S) determined between steps2200and2204can be stored to memory as characteristics of a peak904. The range of values in the peak904can be significant color distribution, in which a significant number of pixels of similar color were found in the accumulated histogram900.

At step2208, the values of the histogram900between the low index level (10) and high index level (hi) can be set to zero, such that the values in the current peak904are not considered again in searches for additional peaks904. The original accumulated histogram900can be saved with its values unaltered for future use.

At2210, the system can determine whether the number of peaks904found in the histogram900is larger than a predetermined target number of peaks. If the number of peaks904is larger than the predetermined target number of peaks, the system can exit at step2212, and use the color ranges determined between the low index level (lo) and high index level (hi) for each peak904as the candidate color ranges of the key1700. If the number of peaks904is less than or equal to the predetermined target number of peaks904, the system can return to step2200to search for another peak900. The system can use the range between the low index level (lo) and high index level (hi) of each peak904to create a candidate range, and the candidate ranges can be used at step1614as the candidate color ranges of the key1700.

At step1616, the system can use each of the candidate color ranges of the key1700determined during step1614to generate a candidate binary key mask2300comprising key pixels2302and non-key pixels2304, with the key pixels2102of each candidate binary key mask2300being those pixels that have color components matching one candidate color range of the key1700and the non-key pixels2104being those pixels that do not have color components matching the candidate color range of the key1700. By way of a non-limiting example,FIG. 23Adepicts a right camera view long shot frame100,FIG. 23Bdepicts a binary mask406of the frame100shown inFIG. 23A, andFIGS. 23C-23Erespectively depict three candidate binary key masks2300each generated from the frame100shown inFIG. 23Ausing different candidate color ranges for the key1700.FIG. 23Cwas generated using a range of hue values between 122 and 152;FIG. 23Dwas generated using a range of hue values between 43 and 62; andFIG. 23Ewas generated using a range of hue values between 153 and 179.

At step1618, the system can perform a shape analysis of each candidate key mask2300. If the key pixels2302of a candidate key mask2300fit into an expected pattern, the candidate color range from which the candidate key mask2300was generated can be confirmed to be the final color range for the key1700. Each candidate key mask can be subject to shape analysis during step1618.

In some embodiments, in order to identify and verify if a candidate key pixel2302does represent a key1700, the system can test if the key pixels2302fall into a selected region2400of the frame100by checking one or more regional pixel ratios. By way of a non-limiting example,FIG. 24depicts the frame100divided into four regions2402: an upper portion2402a, a center left portion2402b, a center right portion2402c, and a lower portion2402d, in which the center left portion2402bcan be the selected region2400on which regional pixel ratio analysis can be performed. By way of another non-limiting example,FIG. 25depicts the frame100divided into four regions2502: an upper left portion2502a, a center left portion2502b, a lower left portion2502c, and a right portion2502d, in which the center left portion2502bcan be the selected region2400on which regional pixel ratio analysis can be performed.

In some embodiments, a horizontal regional pixel ratio (Rh) can be determined by dividing the number of key pixels2302in the region2402bby the number of key pixels2302in both regions2402band2402cfor a left camera view long shot frame. Similarly, a vertical regional pixel ratio (Rv) can be determined by dividing the number of key pixels2302in the region2502bby the number of key pixels2302in the regions2502a,2502b, and2502cfor a left camera view long shot frame. For right camera view long shot frames, the horizontal regional pixel ratio (Rh) and the vertical regional pixel ratio (Rv) can be similarly defined, with the testing pixels being taken from the right half of the frame100instead of the left half as can be done for left camera view long shot frames. The system can accumulate Rh*Rv for each candidate color range based on the candidate key masks2300.

After a predetermined number of candidate key masks2300have been processed during step1618, the candidate color ranges can be sorted based on their accumulated Rh*Rv. If the accumulated Rh*Rv is larger than a predetermined threshold, the candidate color range with the largest accumulated Rh*Rv, can be identified and verified to be the color range that can detect the key area1700. By way of a non-limiting example, the candidate key mask2300bshown inFIG. 23Dcan be verified as depicting the key1700because its key pixels2302pass the shape analysis of step1618, and the color range between the hue values of 43 and 62 used to generate the candidate key mask2300bcan be determined to be the verified color range for the key1700.

At step1620, if the shape analysis and key color range verification of step1618determined that one of the candidate key masks2300showed the key1700, the confirmed color range of the key1700can be output to step220to add and/or update the determined color ranges of the keys1700as playfield detection criteria110. If the shape analysis and key color range verification of step1618did not find that one of the candidate color ranges was a color range for the key1700, for example if no candidate color range had an accumulated Rh*Rv larger than the predetermined threshold, the system can return to step220without updating the playfield detection criteria110, and/or can restart the key color estimation and verification at a later time or stop further trial of key color estimation. In some embodiments, a key mask2300generated with the color ranges for the key1700can be combined with a binary mask406to obtain a combined mask2800representing the entire playfield102. By way of a non-limiting example,FIG. 26depicts a binary key mask2300comprising key pixels2302generated from the color ranges of the key1700confirmed during steps1618and1620, andFIG. 27depicts a binary mask406comprising field pixels402generated during step204.FIG. 28depicts an exemplary combined mask2800with its mask pixels being the key pixels2302and the field pixels402, thereby representing the entire playfield102.

The execution of the sequences of instructions required to practice the embodiments may be performed by a computer system2900as shown inFIG. 29. In an embodiment, execution of the sequences of instructions is performed by a single computer system2900. According to other embodiments, two or more computer systems2900coupled by a communication link2915may perform the sequence of instructions in coordination with one another. Although a description of only one computer system2900may be presented herein, it should be understood that any number of computer systems2900may be employed.

A computer system2900according to an embodiment will now be described with reference toFIG. 29, which is a block diagram of the functional components of a computer system2900. As used herein, the term computer system2900is broadly used to describe any computing device that can store and independently run one or more programs.

The computer system2900may include a communication interface2914coupled to the bus2906. The communication interface2914provides two-way communication between computer systems2900. The communication interface2914of a respective computer system2900transmits and receives electrical, electromagnetic or optical signals, that include data streams representing various types of signal information, e.g., instructions, messages and data. A communication link2915links one computer system2900with another computer system2900. For example, the communication link2915may be a LAN, an integrated services digital network (ISDN) card, a modem, or the Internet.

A computer system2900may transmit and receive messages, data, and instructions, including programs, i.e., application, code, through its respective communication link2915and communication interface2914. Received program code may be executed by the respective processor(s)2907as it is received, and/or stored in the storage device2910, or other associated non-volatile media, for later execution.

In an embodiment, the computer system2900operates in conjunction with a data storage system2931, e.g., a data storage system2931that contains a database2932that is readily accessible by the computer system2900. The computer system2900communicates with the data storage system2931through a data interface2933.

Computer system2900can include a bus2906or other communication mechanism for communicating the instructions, messages and data, collectively, information, and one or more processors2907coupled with the bus2906for processing information. Computer system2900also includes a main memory2908, such as a random access memory (RAM) or other dynamic storage device, coupled to the bus2906for storing dynamic data and instructions to be executed by the processor(s)2907. The computer system2900may further include a read only memory (ROM)2909or other static storage device coupled to the bus2906for storing static data and instructions for the processor(s)2907. A storage device2910, such as a magnetic disk or optical disk, may also be provided and coupled to the bus2906for storing data and instructions for the processor(s)2907.

A computer system2900may be coupled via the bus2906to a display device2911, such as an LCD screen. An input device2912, e.g., alphanumeric and other keys, is coupled to the bus2906for communicating information and command selections to the processor(s)2907.

According to one embodiment, an individual computer system2900performs specific operations by their respective processor(s)2907executing one or more sequences of one or more instructions contained in the main memory2908. Such instructions may be read into the main memory2908from another computer-usable medium, such as the ROM2909or the storage device2910. Execution of the sequences of instructions contained in the main memory2908causes the processor(s)2907to perform the processes described herein. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions. Thus, embodiments are not limited to any specific combination of hardware circuitry and/or software.

Although the present invention has been described above with particularity, this was merely to teach one of ordinary skill in the art how to make and use the invention. Many additional modifications will fall within the scope of the invention, as that scope is defined by the following claims.