Patent ID: 12217771

DETAILED DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS

Some example embodiments of the present invention include systems and procedures which may condense video recordings with long sequences of video frames into a representation of pixels of arbitrary length and height. In example embodiments, such video recordings can be any compressed, e.g. MPEG, or uncompressed digital video recording. In some examples, this condensed representation can be used as a visual and navigational aid in systems for analyzing video recordings by representing the characteristics of the video that would help in detecting scene or material changes in the content or in systems that perform frame accuracy analysis. In examples, such representations may be used in systems that perform video editing or any application that displays a large amount of visual information.

Example embodiments of the present invention include systems and procedures which may include generating a condensed representation by condensing each video frame of the recording in a horizontal dimension to a relatively narrow width, e.g., one pixel wide. In examples, the video frame may also be condensed in the vertical dimension to a height that may be, e.g., several pixels high. In one preferred example, each condensed video frame may have a width that is less than its height. Additionally, to achieve highly condensed representations, in some examples a frame may be condensed to a representation that is smaller than one pixel wide, e.g., multiple frames can be condensed into a single column of pixels. An example procedure100for generating a condensed representation of a video is presented inFIG.1. The example procedure illustrated inFIG.1begins with a video frame101, for example a 720×480 pixel video frame, which is resized to a condensed representation103, for example, a 1×20 pixel representation. It is noted that the following example is discussed making reference to a 720×480 video frame. The present invention is, however, not limited to video frames of any particular size and the example used herein is for illustration purposes only. The example procedure illustrated inFIG.1may be performed in two steps. For example, the example procedure may begin with a video frame101, which may be an array of pixels. In the illustrated example, the720dimension may represent the rows of the array and the 480 dimension may represent the columns of the array. In an example procedure, the first step, in generating a condensed representation103, may include averaging the color of each pixel in every row producing a new pixel array102of, e.g., 1× 480 pixels. The color of each pixel in the new pixel array may be the average of the color of the pixels of the row. Example procedures may proceed to further condense the representation in the other dimension. For example, an example procedure may average the color of the pixels in the 480 dimension illustrated inFIG.1. In the example procedure shown inFIG.1, the 480 pixels may be grouped into twenty consecutive 24-pixel blocks. In an example, the average color of the pixels in each block may be calculated and used to represent the block. For example, as shown inFIG.1, the 1×20 pixel representation,103, may be an aggregation of the twenty average colors that correspond to the twenty blocks.

It can be understood that the choice of breaking the 420-pixel dimension into 24-pixel blocks may be related to, e.g., the actual frame height, the amount of color information and detail the user chooses to include in the representation, as well as, the characteristics of the actual video recording. A smaller number of blocks may provide a more condensed representation and less color information. Therefore, to get a useful and meaningful condensed representation, for example, for a full high-definition recording with a frame size of 1920×1080 pixels, it may be desirable to use a greater number of blocks as compared to a recording with a frame size of 720×480 pixels. Additionally, the choice of the number of blocks may be dictated by the actual recorded content. For example, a recording with subtle scene changes and transitions, can require a greater number of blocks to provide a meaningful condensed representation compared to another recording of the same frame size, but with abrupt and distinguishable scene changes. Example embodiments, therefore, may provide a user with the option to choose among preset or user-defined block sizes. In addition, example embodiments of the present invention may automatically determine an appropriate number of blocks according to specific recording characteristics, e.g., frame size. In other examples, a user may change the number of blocks, for example, with a slider control, to generate a useful representation with appropriate color information and granularity.

In example embodiments, a different number of blocks can be selected and blocks of unequal sizes can be used. For example, video recordings tend to be focused on the center of the frame. In some examples, accordingly, an uneven distribution of pixels in the blocks may be used to produce a more meaningful condensed representation compared to one that uses equal numbers of pixels in the blocks. A user may select, for example, to have small-sized blocks in the middle of the frame and blocks with a greater number of pixels at the top and bottom parts of the frame.

In addition, although the example illustrated inFIG.1may use pixel color averaging to generate a pixel that represents a large number of pixels, in other examples, different and more complex aggregation algorithms than color averaging can be used to represent a number of pixels with a single pixel. In such examples, any algorithms which may condense an entire frame into a “sliver” can be used to generate the condensed representation. In addition, different algorithms may be used to emphasize different characteristics of a frame, for example, visual information of the background, or detail in the center of the frame.

FIG.2presents an example procedure200of generating a condensed video recording representation. In the example procedure, the video frames210, may correspond to a video recording. To generate the condensed video recording representation, initially a condensed representation of each frame220may be generated, and then all the frame representations may be “stacked” side-by-side to produce a horizontally-oriented, condensed bit map representation230of the whole video recording. The pixel strip on the left most side240may correspond to the first frame and the pixel strip on the right most side250may correspond to the last frame. In this case, the horizontal dimension of the representation may capture vertical movement in consecutive frames. Alternatively, in other example procedures, each video frame may be condensed in the vertical dimension. In such example procedures, the condensed representations may be “stacked” in the vertical dimension to form a vertically-oriented representation of the video recording. In such an example, the vertical dimension of the representation may capture horizontal movement in consecutive frames. In example embodiments, condensed representation created may allow the temporal information of the recording, which is captured in multiple consecutive frames, to be visualized in one dimension, i.e. the horizontal dimension or the vertical dimension, as the case may be.

Example embodiments of the present invention may also include systems for generating a condensed representation of a video.FIG.8illustrates an example system800in accordance with an embodiment of the present invention. Such example systems may, for example, have an input device801capable of receiving a video input. Suitable input devices may include networking hardware, hardware to access a local or remote storage device, etc. Using such an input device, example systems may be able to receive video from a variety of sources804, for example, as illustrated inFIG.8, example systems may be able to receive video from video storage systems, from cable networks, and from video editing devices, as well as other sources of video. Example systems may also include a storage device for storing the received video, such as, for example, a magnetic or optical storage system.

Example systems may also include a processor803which may be configured to processes the video into a condensed representation, as described herein. For example, an example system may be configured to receive and store an input video signal. The example system may be able to generate a condensed representation of the video in a first dimension, for example the horizontal dimension. Example systems may also be configured to further reduce the representation created in another dimension, e.g., the vertical dimension. Once created, example systems may store the condensed representation. Example systems may also include an output device801, which may or may not be the same device as the input device referenced above, and may be capable of outputting the condensed representation using such a device.

Example systems may also include a user interface805. For example, such systems may include interfaces805which may allow users to control the creation of condensed representations. For example, example systems may include display hardware, or may include a web, or other, interface through which users may access the system. Example systems may provide users with the ability to control the condensing process through the user of such an interface805, including the ability to choose the video to be condensed and, the manner in which the video is to be condensed, e.g., the number of blocks to use, etc. Such example systems may be capable of displaying the resulting condensed representation to users through the interface805, and may be capable of accepting user input as to any of the other actions described herein.

In addition, other example embodiments of the present invention may provide for systems which may utilize condensed representations of video. As explained further below, such example systems may include video production and editing systems, video consumption systems, surveillance systems, and any other systems in which it may be desirable to navigate through video using a condensed representation. Of course such example systems may or may not be distinct from the systems which generate condensed representations from video.

For example,FIG.3shows an example display of an example system utilizing a condensed representation of a video as discussed above. InFIG.3the example display depicts a 1024×20 bitmap condensed representation310of a three minute recording. Such an example recording may include 5396 frames and each frame may be a 720×480 pixel image. To produce the 1024×20 bitmap representation, initially a 5396-pixel wide and a 20-pixel high representation may have been generated, according to the example method described above. In the illustrated example, that representation may be resized to be 1024-pixels wide and 20-pixels high; which may mean that every 1×20 pixel representation of the bitmap corresponds to a little more than five video frames of the video recording in the example. In example embodiments, such resizing may be performed so that the condensed representation can fit conveniently on a screen. In such example embodiments, any standard resizing technique can be used. The information after condensing and resizing may help a user quickly distinguish scene transitions and other interesting changes in the source material. Of course, it is to be understood that the size of the condensed representation may be changed, as desired, to adjust visibility and according to user preferences.

The example display illustrated inFIG.3, depicts 5 points (A, B, C, D, and E) identified in the condensed representation310.Point A shows a transition from a segment with red hues, to a segment with blue hues, separated by three black frames.Point B shows a similar transition, but the color differences are more subtle. Still, it's possible to detect a transition.Point C shows a scene cut, which is also evident in the strip chart.Point D shows black video frames followed by start of a non-black video sequence. The transition from the black frames to the non-black ones is evident in the strip.Point E also shows black video frames followed by start of a non-black video sequence.

As illustrated, example systems according to the present invention, may also display frames associated with points identified in a condensed representation. For example, as shown inFIG.3seven frames may be displayed for each point. In such example systems, a frame corresponding with an identified point may be displayed, for example, the boxed frame illustrated in the middle of each grouping inFIG.3. In addition, the example system may also show frames surrounding the marked location. InFIG.3, the central frame is depicted along with three frames preceding the identified frame and three frames following it. Such example systems, therefore, may allow users to quickly identify points of interest in a video through use of the condensed representation. Once such points of interest have been found, example systems may allow users to identify those points and may display the corresponding full frame from the video. In addition, example systems may allow users to see those frames in context by displaying a number of frames surrounding the frame of interest.

In example embodiments of the present invention, a system that implements the condensed representation described above may be used for example to analyze TV advertisements within broadcast TV programs. Typically advertisements appear in 30 or 60 second lengths and usually come along with distinguishable scene transitions that can be easily detected using the condensed representation disclosed. Additionally, example embodiments of the present invention may allow such condensed representations, or bitmaps, to be tagged with markers that would point to different interesting events within the recording that a user can quickly identify and to which the user may navigate. Other example embodiments may use the disclosed representation, for example, in systems for scene change analysis and scene prediction, in scene cut detection algorithms, in surveillance video tools, and in video editing systems as a navigation aid. In other example embodiments, the procedures and systems described herein may also be used in systems that perform automated functions without human supervision. For example, in example embodiments, when computing the condensed frame representations, each frame representation may be appended with corresponding metadata. Such metadata may include representative parameters of the frame, i.e. an average color of the pixels of the frame, a standard deviation from the average color, or any other characteristic and statistical parameters. In some examples, the metadata may also include time stamps specific to each frame and other information related to each corresponding frame to which they are appended. Embodiments of the present invention may utilize such metadata. For example, an example system may check for unexpected and/or unpatterned changes in such metadata, and may automatically detect unusual and/or anomalous events within a recording and may insert a flag marking such an event or may draw a user's attention to the event, e.g., in a surveillance review system.

Referring again to the example display ofFIG.3, the condensed bitmap and actual video frames can be viewed as a webpage using a standard web browser, although the representation can also be viewed within a desktop application or any application that can display video. In example embodiments, a user can click at a point on the bitmap, e.g. point A, and the actual frame that corresponds to the clicked point may be presented. Specifically, in the example, a sequence of, for example, seven frames may be displayed, which may include the frame that corresponds to the point selected on the representation and the three preceding and three following frames, as described above. In example embodiments, the number of preceding and following frames displayed may be any suitable number. In some examples, the number may be a parameter that may be changed by a user, so that the display provides appropriate local content around the clicked point.

In example embodiments of the present invention, systems and methods are provided which may allow users to identify scene changes in video, utilizing the example condensed representations discussed herein. For example,FIG.4shows condensed representations400of four different video recordings. These representations show that in a sequence of video segments, related video tends to change slowly over time. However, when a scene cut or a change of content appears in video, there is usually an abrupt change in the luminosity and the brightness of the video image. For example, inFIG.4, some of the transitions occurring in the four condensed representations are marked with arrows. Such abrupt changes may be clearly distinguishable in condensed representations of the video. Example embodiments of the present invention may exploit such condensed representations allowing a user to detect scene cuts.

In addition, such example embodiments may also allow users to detect other features of a video. For instance, some movement that occurs over time may be detected. InFIG.4, towards the end of the fourth strip410, a portion of the strip has been exploded420. It can be assumed from the condensed representation that the video depicts a large object which moves from the top part of video to the middle. Looking closely from left to right, within the exploded portion430, it seems that the object is falling.

An example embodiment of the present invention, namely a video insertion accuracy tool provided for use in a cable television system, will now be discussed. It is, however, to be understood that such example embodiments may be capable of use in any system that uses or requires video analysis and that the present invention is not limited to either a cable network environment or the or to insertion accuracy tools.

As mentioned, some example embodiments of the present invention may provide for a video insertion accuracy tool, which may allow a user to analyze the accuracy with which content is inserted into video programming. Video insertion accuracy tools may be used in television-based systems, for example, cable or satellite systems, in online applications, and, in general, in any application and system where frame accuracy is important. In cable systems, advertisements can be inserted locally at the different cable markets. Local advertisement content can be inserted in the cable broadcast transport stream, when indicated by the cable network, for example, upon reception of a cue tone from the cable network. In practice, local advertisement insertion is an inherently inaccurate process requiring constant fine tuning by multi-system operators (MSOs) of cable television systems. In such situations, an example video insertion application tool, in accordance with example embodiments of the present invention, may be installed at such MSO locations, for instance in a cable television headend facility. Example tools may be capable of automatically recording a program for analysis. Such example system may automatically record video surrounding the insertion of local content. For example, example system may identify and record indications received from a cable network identifying appropriate insertion times of local advertisements, which are typically provided ahead of the intended times of insertion in a cable system. Example tools may receive these indications and may automatically record all of the content necessary to analyze the insertion accuracy of the locally inserted content, including both the video and the indications themselves.

Local ads are most commonly inserted into a transport stream using stream splicing. In such cases, the spliced stream may then be fed into an analysis device, in accordance with an example embodiment of the present invention. The video insertion accuracy tool may receive the output of the splicing device to record the locally inserted ads. In addition to the digital video streams, the example tool may also receive cue tones provided by the cable network which signal an upcoming local availability for an advertisement insertion. Upon reception of a cue tone, the tool may start recording the audio and video content of that particular network stream to a storage device. Typical advertisement breaks last several minutes and the recordings may cover the entire break. There is a specific agreed-upon delay between the indication coming from the network and the time when the local insertions are supposed to start, so that the insertion is synchronized. This delay is called a “preroll.” Unfortunately, the preroll is not always accurate and may change, which causes synchronization problems. Different cable television systems may use different types of indication tones. It is to be understood that the example tools described are not indication tone specific, but can work with any type of indication tone and, therefore, different cable television systems.

Once such information is gathered, example insertion analysis tools may allow users to examine insertion accuracy in the content. In some example embodiments, after the recordings are stored, the example tool can display a list of all the available recordings for further analysis. For example,FIG.5displays a screenshot500of available recordings that an example tool may display to a user. In example embodiments, the users may be able to select a specific recording from such a screen for further analysis. Timing information related to a selected recording may be populated by the system to aid the analysis. An operator may confirm the results of the analysis and also make the needed corrections in the recording. As illustrated inFIG.5, example systems may display other relevant information allowing users to identify and analyze video which the system records. For instance, in the illustrated example, users may be presented with the file name of the stored recording, the zone ID, Channel ID, Channel label, Date of the recording, the time of the cue tone, the preroll, the time at which the system began to record the video segment, the duration of the recording, and a field indicating whether the particular recording has been analyzed.

After selecting a recording, in example embodiments, a user may be presented with a screen similar to the screenshot600shown inFIG.6, which shows a full-screen representation of a frame610, a navigation control which may allow a user to navigate within a video recording or a portion of the recording being displayed620, and a visual representation of the entire recording630according to embodiments of the present invention. In some examples, the full-sized frames can be decoded and displayed on the fly.

Some example tools may indicate when actual insertion points take place, using timing information provided by a cable network or received from another source. Insertion errors exist when there is a difference between the intended and the actual insertion point. An example of an insertion error is shown inFIG.6, where the actual insertion point650is two frames late from the intended insertion point640. It is also possible that the actual insertion point may be earlier than intended. Example embodiments may display insertion error information. For example,FIG.6illustrates a screen of an example tool which provides additional information660, e.g. frame number, insertion time, and insertion error. For example, in the example screen show inFIG.6the intended insertion frame number is 883 and the actual insertion frame number is 885, thus, there was a two-frame delay, which corresponds to an insertion error of 0.067 seconds.

The condensed visual representation630, illustrated in theFIG.6example, which looks like a bar at the bottom of the screenshot, may help a user identify where scene cuts take place and may make it possible to quickly identify the different advertisements, or other inserted content. Scene transitions between different advertisements are easily observed, because advertisement content typically has a distinct look. Therefore different advertisements may be clearly distinguishable in the condensed representation of the recording, which may help a user navigate to the scene transitions. In example embodiments, a user may click on the condensed video representation630, and may then preview the nearest frame that corresponds to the point clicked. In some examples, the condensed video representation for each video recording may be precomputed, according to the example processes described above, once recording is complete. In the example shown inFIG.6, each stack of pixels in the representation bar represents a frame. The example representation may also be scaled to fit a certain width. For example, the condensed representation for the example recording ofFIG.6may have first been created to be 5396-pixels wide and 20-pixels high. The condensed representation may have then been resized to 1024×20 pixels as shown630inFIG.6. In example embodiments the condensed representation need not be displayed as a single line. Rather, the representation may be arranged in different ways, e.g. it can be arranged in several rows, for example, for a very long recording, or it can be arranged in one row as inFIGS.3,4, and6. Generating the condensed representation and resizing it to fit a certain width may be performed in a two-step process as was described previously. However, it is to be understood that in other examples condensing the video sequence and resizing the condensed representation can be performed in a single step.

Example insertion analysis tool interfaces may allow for quick identification of insertion errors, like the two-frame insertion delay error discussed with regard toFIG.6. Example embodiments may allow users to record the results of an analysis. Once the results for multiple recordings are collected, they may be further analyzed by the user. By measuring and tracking insertion errors over time, certain types of problems, for example, inaccuracies at a cable network, inaccuracies in an MSOs transport path, inaccuracies in the configuration of splicing devices, etc., can be identified. Example embodiments of the present invention may provide users with the tools necessary to conduct such an analysis. For example,FIG.7shows an example analysis display which may be provided by example embodiments of the present invention, which may permit users to analyze and view the results from the analysis of different recordings. Specifically, the screenshot700inFIG.7may illustrate an example tool which may allow users to graph the insertion inaccuracy for different selected networks at different days. Example embodiments may allow users to track changes in accuracy, like the ones shown, over time which may also be correlated to different events. For instance, if all networks at a particular moment fluctuate by the same insertion inaccuracy, then that points to a problem in the MSO transport infrastructure. Example embodiments of the present invention may utilize information collected about the inaccuracies at the insertion points to help guide corrective measures. Example tools may allow users to view the results of individual analyses in a variety of ways. For instance, the example shown inFIG.7, allows users to choose a select a time period for analysis and also select a number of networks about which data is to be presented. In the example, a user has elected to view a graphs of the period from 212212008 to 2/28/2008, and has chosen to present information related to the Nickelodeon, CBS, Comedy Central, and ESPN networks. Example systems may then generate and display a graph, such as the example graph illustrated, which may allow a user to identify insertion errors for the various networks, or other video sources, identified, as well as allowing users to compare errors between the sources and to view the change in various errors over time.

An example system implementing an insertion analysis tool900is illustrated inFIG.9. As shown, such an example system may be located within a905cable television system, or other system which provisions video. Such a system may include an I/O device901which may be in communication with a video source, for instance, in the example the system is connected to a splicing device906which may provide video to the system. Example systems may also include a storage device902which may be capable of storing relevant video and other material, e.g., signals from the cable system that content should be inserted. Example systems may also include one or more processors903. Such processors may generate a condensed representation of the video, as described above. Alternatively, example systems may receive a condensed representation as an input to the system. In example systems, the processor903may be configured to facilitate the analysis of insertion errors described more fully above. In addition, example systems may provide user interfaces905. For example, example systems may allow users to access the services provided by the system using a web browser. Such users may then be able to access the analysis tools described above, and other useful tools. For example, users may be able to choose video segments to analyze, may manage the recording of video segments, may view video segments and condensed representations of those segments, may perform analysis of those segments, and may view statistical analyses such as the graph presented inFIG.7.

It will be appreciated that all of the disclosed methods and procedures described herein can be implemented using one or more computer programs or components. These components may be provided as a series of computer instructions on any conventional computer-readable medium, including RAM, ROM, flash memory, magnetic or optical disks, optical memory, or other storage media. The instructions may be configured to be executed by a processor, which when executing the series of computer instructions performs or facilitates the performance of all or part of the disclosed methods and procedures.

It will further be appreciated that the above-described methods and procedures may be provided using the systems disclosed herein, or on other types of systems. The methods and procedures, unless expressly limited, are not intended to be read to require particular actors or systems performing particular elements of the claimed methods.

In the preceding specification, the present invention has been described with reference to specific example embodiments thereof. It will, however, be evident that various modifications and changes may be made thereunto without departing from the broader spirit and scope of the present invention as set forth in the claims that follow. The specification and drawings are accordingly to be regarded in an illustrative rather than restrictive sense.