Using speech to text for detecting commercials and aligning edited episodes with transcripts

Methods and apparatus, including computer program products, for using speech to text for detecting commercials and aligning edited episodes with transcripts. A method includes, receiving an original video or audio having a transcript, receiving an edited video or audio of the original video or audio, applying a speech-to-text process to the received original video or audio having a transcript, applying a speech-to-text process to the received edited video or audio, and applying an alignment to determine locations of the edits.

BACKGROUND OF THE INVENTION

The invention generally relates computer systems and computer executed methods for using speech to text for detecting commercials and aligning edited episodes with transcripts.

In general, video clips are short clips of video, usually part of a longer recording. If video clips originate from broadcast video content (e.g., over-the-air, cable, satellite, and so forth), there is frequently closed captioning associated with the broadcast and commercials. In general, closed captioning is the process of displaying text on a television, video screen or other visual display to provide additional or interpretive information to individuals who wish to access it. Closed captions typically show a transcription of the audio portion of a program as it occurs (either verbatim or in edited form), sometimes including non-speech elements.

Making video clips on the web, on smart phones, and so forth, matched up to the relevant closed caption text is less expensive than human transcription and yields better results than purely automated speech-to-text methods as the closed caption files were generated by a human. However, typically the closed caption will not exactly match the spoken words, it is usually quite different as the closed captioner focuses on important words, s/he makes mistakes, and so forth.

The closed caption also lags the broadcast video as the close captioner needs to watch/hear the video and then input the corresponding closed caption. This lag varies. For pre-recorded (as opposed to live) content, there may be no lag at all because the lag was already edited out. If one uses an automated technique, such as speech-to-text, to generate words from the video clip to assist in an alignment process, there will often be recognition errors. A variability in lag along with the errors in both the closed caption text and speech-to-text make alignment complicated.

In addition, many media broadcasters do not have the closed caption text readily available, so frequently, one needs to capture closed captions from a live broadcast stream.

Further, sometimes commercials are closed captioned and other times they are not. Thus, if a media company desires to take an edited version of a broadcast, with commercials removed, and align it with the original broadcast having commercials, time gaps and whether the commercials are closed captioned or not present many complications.

SUMMARY OF THE INVENTION

The present invention provides methods and apparatus, including computer program products, for using speech to text for detecting commercials and aligning edited episodes with transcripts.

In general, in one aspect, the invention features a method including, in a computer system having a processor and a memory, receiving an original video or audio having a transcript, receiving an edited video or audio of the original video or audio, applying a speech-to-text process to the received original video or audio having a transcript, applying a speech-to-text process to the received edited video or audio, and applying an alignment to determine locations of the edits.

DETAILED DESCRIPTION

As shown inFIG. 1, a system10includes a processor12and memory14. Memory14includes an operating system (OS)16, such as Linux®, Snow Leopard® or Windows®, and a process100for using speech to text for detecting commercials and aligning edited episodes with transcripts. The transcripts may come from the closed-captions or from other sources. While we focus the discussion on transcripts and closed-captions, the present invention can also be applied to other metadata associated with the video or audio, for example, director's commentary, scene descriptions, and so forth. The system10may include a storage device18and a communications link20to a network of interconnected computers22(e.g., the Internet). Process100, fully described below, executes speech to text on both an original un-edited video and the edited video. Then by aligning and comparing recognized word outputs, process100can identify which segments exist in the original un-edited video but not in the edited video. Though the editing may change the exact words recognized at each time location, there is enough similarity in the overall recognition to perform this match.

It should be noted that closed captions as described herein can be any source of word transcription. Further, a commercial can generally refer to other segments deleted from the edited video.

As shown inFIG. 2, the process100for using speech to text for detecting commercials and aligning edited episodes with transcripts includes receiving (102) an original video or audio having a transcript. Process100receives (104) an edited video or audio of the original video or audio.

Process100applies (106) a speech-to-text process to the received original video or audio having a transcript.

Process100applies (108) a speech-to-text process to the received edited video or audio.

Process100applies (110) an alignment to determine locations of the edits.

Process100aligns (112) an output of the speech-to-text process to the received original video or audio having a transcript and the output of the speech-to-text process to the received edited video or audio. The alignment (112) a text alignment can using various techniques, such as, for example, dynamic programming to minimize a metric calculated from the number of substitutions, insertions, deletions.

Process100examines (114) alignments for large sections that exist only in the original and not in the edited version. Those are the detected edits (typically commercials). This can involve looking at a sliding window centered around a putative commercial start/end point and considering the accuracy within the window to the left verses to the right of the putative start/end point. The accuracy can also take into account confidence scores from the speech to text output. Process100can also have minimum/maximum length restrictions on detected commercials.

A finite state machine can also be used to assist, for example, process100might look for multiple potential commercial start boundaries before declaring that a start boundary actually exists. More simply, process100would not allow a commercial start unless it is not already in a commercial, conversely, process100might not allow a commercial end unless it is already inside a commercial.

Looking at the time stamps to the left and right of the putative commercial can be very effective in filtering out false positives (because the timeline must include removed material).

Rule based logic involving multiple of these criteria can be used for a more robust determination. Similarly, a statistically trained classifier or other machine learning system might be used.

For each pair of segments of the remaining segments aside from the commercials in the original, and the corresponding segments in the edited version, process100executes (116) a closed-caption alignment process200.

It should be noted that when text is aligned, process100involves correct matches, substitutions, insertions and deletions. The following is one example.

Reference: The rain in Spain stays mainly in the plains

Hypothesis: rain near Spain stays mainly mostly in the plainsD C S C C C I C C C

where C=Correct, S=Substitution, I=Insertion, and D=Deletion, all relative to the reference text. In the hypothesis text above, “The” is deleted, “near” is substituted for “in,” “mostly” is inserted and the remaining text is correct.

As shown inFIG. 3, the closed-caption alignment process200includes receiving (202) an edited video clip to be aligned with closed captions along with a start time. Closed captions can include subtitles, subtext, a summary that is approximately contemporaneous with the words spoken, and verbal descriptions of what's happening in the video but not actually spoken.

Process200receives (204) an original video clip associated with closed captions.

Process200determines (206) the locations of the edits.

Process200determines (208) a candidate closed caption text for the edited video clip by taking the closed captions from the original video clip and considering the locations of the edits.

Process200executes (210) a speech-to-text process on the edited video clip.

Process200executes (212) a speech-to-text process to the candidate closed caption text.

Process200executes (214) a best alignment of the candidate closed caption text output to the speech-to-text output. The best alignment can use one of numerous techniques, such as, for example, dynamic programming algorithms to minimize the number of mismatches. Certain error types may be considered worse than others in a scoring. Some words or word patterns might may be given a penalty of zero if, for example, they are likely to be omitted from the closed caption.

Sequences within the alignment that “match well” form islands. Matching well may be determined by criteria including, for example, contiguous matches, contiguous matches except for certain types of errors like insertions and deletions, allowing for a small ratio of errors, such as up to one in every four contiguous words, and so forth. Alternatively, process200can rank the sentences by the number of matching words in each sentence and consider the top N to form islands, where M may be 1. It should be noted that “sentence” may not necessarily be a linguistic sentence, but may be one selected by the speech to text process (e.g., a pause during speech may be labeled as a sentence boundary by speech to text).

Process200identifies (216) likely boundaries using the islands. For example, consider a 10 minute video clip. If the largest island corresponds to time 1 m 0 s to 8 m 0 s (2 m 0 s from the end) in the video clip and time 2 m 0 s to time 9 m 0 s in the candidate closed caption file, the likely start/end boundaries are 1 m 0 s and 11 m 0 s in the closed caption file (because we started 1 m 0 s from start of clip and ended 2 m 0 s from end of clip, so those offsets are added to the island.

Process200can perform one or more passes in identifying islands and boundaries, and may add some padding to the boundaries to allow for slight errors. Process100may use either word or sentence time stamps from the speech-to-text transcript for boundaries identification.

In some passes, process200may take boundaries identified in a previous pass, and move the end points identified in a previous pass by finding very near islands matching a minimum criteria, as opposed to maximizing the island size. For example, in pass1, process100may find the largest island to infer candidate end points, and then in pass2, process100may adjust the end points by finding the location of the nearest three word match. This three word match may permit skipping over certain errors, such as insertions and deletions. Thus, a three word match excluding insertions and deletions becomes the minimum criteria in this example.

A common pattern is that a headline is repeated twice, both as an introductory prelude, and later in the video clip itself. For example, the closed caption may have “Coming up next: Does it pay to play it safe in this economy? <intervening words> We asked the question does it pay to play it safe in this economy? . . . ” with the actual clip starting at “We asked the question . . . ” Multiple passes are particularly useful for eliminating the introductory prelude in the first pass, and later to refine the location of the boundaries.

Once process200has determined a final set of candidate end point boundaries, process200takes the corresponding portion of the closed caption file and outputs (218) final closed captions aligned to the video clip.

Process200may further perform a sentence level or word level realignment of a final portion of the closed caption file and the video clip to output a fully aligned closed caption to the video clip data structure.

If the final alignment error rate is too high or if any of the passes should fail, process200can label the alignment as having failed instead of returning a low quality alignment.

In another embodiment, speech-to-text to replaced by audio fingerprinting, e.g., a reduced-bit representation or other identifier of a media signal (e.g., audio, video or still images). One such method includes receiving a partial fingerprint of a media signal, wherein the partial fingerprint is derived from the media signal itself, the partial fingerprint being computed a first location, processing the partial fingerprint to obtain a more refined fingerprint, relative to the partial fingerprint, the act of processing occurring at a second location that is remote from the first location, and based at least in part of the refined fingerprint, identifying the media signal or information associated with the media signal.

The foregoing description does not represent an exhaustive list of all possible implementations consistent with this disclosure or of all possible variations of the implementations described. A number of implementations have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the systems, devices, methods and techniques described here. Accordingly, other implementations are within the scope of the following claims.