Stream synchronization using an automated digital clapperboard

An approach is provided in which the approach receives a media stream comprising an audio stream and a video stream. The audio stream includes a set of audio tones inserted at a set of predetermined frame intervals, and the video stream includes a set of predetermined colors inserted into a set of predetermined pixel locations at the predetermined frame intervals. The approach determines that an audio frame in the audio stream at which the set of audio tones are inserted is misaligned in time with a video frame in the video stream at which the set of predetermined colors are included in the set of predetermined pixel locations. The approach synchronizes the audio stream with the video stream in response to the determination and aligns in time the audio frame with the video frame at one of the predetermined frame intervals.

BACKGROUND

Media content streaming is typically a process by which a content distributor provides media content to a content receiver over some type of network connection, such as a satellite channel, a cable channel, or the Internet. A content distributor may be an entity that distributes media content, such as a television station, a streaming Internet channel, a video streaming service, etc. A content receiver may be a system, device, or module that receives the media content at a user's location and provides the media content to the user.

Streaming media is multimedia that is constantly received by and presented to an end-user. Live streaming is the delivery of Internet content in real-time similar to live television broadcasts. Live streaming requires a form of source media (e.g. a video camera, an audio interface, screen capture software), an encoder to digitize the content, a media publisher, and a content delivery network to distribute and deliver the content. A webcast is a live or on-demand presentation streamed over the Internet in audio or audio and video. Webcasting is essentially “broadcasting” over the Internet and is ideal for targeting large audiences.

BRIEF SUMMARY

According to one embodiment of the present disclosure, an approach is provided in which the approach receives a media stream comprising an audio stream and a video stream. The audio stream includes a set of audio tones inserted at a set of predetermined frame intervals, and the video stream includes a set of predetermined colors inserted at a set of predetermined pixel locations at the predetermined frame intervals. The approach determines that an audio frame in the audio stream at which the set of audio tones are inserted is misaligned in time with a video frame in the video stream at which the set of predetermined colors are included in the set of predetermined pixel locations. The approach synchronizes the audio stream with the video stream in response to the determination and aligns in time the audio frame with the video frame at one of the predetermined frame intervals.

DETAILED DESCRIPTION

ExpressCard155is a slot that connects hot-pluggable devices to the information handling system. ExpressCard155supports both PCI Express and Universal Serial Bus (USB) connectivity as it connects to Southbridge135using both the USB and the PCI Express bus. Southbridge135includes USB Controller140that provides USB connectivity to devices that connect to the USB. These devices include webcam (camera)150, infrared (IR) receiver148, keyboard and trackpad144, and Bluetooth device146, which provides for wireless personal area networks (PANs). USB Controller140also provides USB connectivity to other miscellaneous USB connected devices142, such as a mouse, removable nonvolatile storage device145, modems, network cards, Integrated Services Digital Network (ISDN) connectors, fax, printers, USB hubs, and many other types of USB connected devices. While removable nonvolatile storage device145is shown as a USB-connected device, removable nonvolatile storage device145could be connected using a different interface, such as a Firewire interface, etcetera.

As discussed above, webcasting is a live or on-demand presentation streamed over the Internet. A challenge found, however, is that at times the webcast's video signal at the receiving device is not properly synchronized with the webcast's audio signal due to, for example, transmission delays in the network and/or unequal receiving device processing delays.

FIGS. 3 through 10depict an approach that can be executed on an information handling system that resolves the aforementioned challenges by injecting human inaudible sounds into the audio stream in conjunction with injecting human unperceivable visual patterns in the video stream to automatically re-sync the two streams at a receiving device.

In traditional film making and video production, producers use a ‘clapperboard’ to assist synchronizing picture with sound and to designate and mark the various scenes as they are filmed and audio-recorded. The clapperboard combines a chalkboard or digital slate with filmsticks that, when shut, generate a sharp “clap” noise that is easily identified on the separate audio track. The audio and video tracks can later be precisely synchronized by matching the sound and movement in the video. Since each take is identified on both the visual and audio tracks, segments of film are easily matched with segments of audio. As discussed below, the approach discussed herein digitally injects “audio claps” into the audio stream and “video claps” into the video stream at specific frame intervals that a receiving device utilizes to synchronize the audio stream with the video stream.

FIG. 3is an exemplary diagram that depicts a transmitting system injecting clapperboard sets into a media stream and a receiving device synchronizing the audio and video portions of the media stream using the clapperboard sets (audio clap and corresponding video clap). Transmitting system300receives or generates media stream305, which includes audio stream310and video stream315. Digital clapperboard injector320selects a frame interval, a random set of inaudible tones for audio claps, along with a random set of colors and a random set of coordinates for video claps. In one embodiment, digital clapperboard injector320uses a random number generator to select the clapper values. For example, digital clapperboard injector320may use the random number generator to select four audio tone values from 1-20 and 20,000-22,000, and then use the random number generator to select color values between 0-100 for red, green blue (RGB) as well as x/y pixel coordinates between 0-1000 (seeFIG. 9and corresponding text for further details).

Transmitting system300then sends synchronized clap injected media stream325over computer network345to receiving device350. At some point, clap injected audio stream335and clap injected video stream340become unsynchronized with each other (unsynchronized clap injected media stream355).

Digital clapperboard synchronizer360uses the clapperboard set information in both clap injected audio stream335and clap injected video stream340as discussed herein to synchronize the two streams and eventually produce synchronized media stream365, which includes audio stream370(same as audio stream310) and video stream375(same as video stream315) that receiving device350presents to a user (seeFIGS. 6, 8, 10, and corresponding text for further details).

FIG. 4is an exemplary diagram that depicts a clap injected media stream becoming unsynchronized as it traverses through a computer network. Clap injected audio stream335includes audio claps A1, A2, A3, A4, and A5 injected at the predefined frame interval (e.g., every 500 frames). Clap injected video stream340includes corresponding video claps V1, V2, V3, V4, and V5 also injected at the predefined frame interval. At this point, clap injected audio stream335is synchronized with clap injected video stream340.

When synchronized clap injected media stream325passes through computer network345, the audio stream and video stream become unsynchronized (unsynchronized clap injected media stream355). As can be seen, clap injected audio stream335's audio claps A1, A2, A3, A4, and A5 are ahead in time of clap injected video stream340's video claps V1, V2, V3, V4, and V5. As such, as discussed below, digital clapperboard synchronizer360will delay clap injected audio stream335until clap injected video stream340catches up and the two streams become synchronized.

FIG. 5is an exemplary diagram that depicts a digital clapperboard injector injecting digital claps into both an audio stream and a corresponding video stream. Transmitting system300uses digital clapperboard injector320to inject audio claps into media stream305's audio stream310in synchronicity with injecting video claps into media stream305's video stream315.

Digital clapperboard injector320stores preselected audio tones510, preselected frame interval530, preselected colors550, and preselected pixel coordinates560, each of which is selected by a user or a random number generator. Metadata generator500captures each of the preselected values and generates metadata header330, which is sent to receiving device350via clap injected media stream325.

Frame counter540receives preselected frame interval530and instructs both audio clap injector520and video clap injector570to inject their respective claps into their respective audio stream310and video stream315at the predetermined frame intervals (e.g.,500,100,1500,2000, etc.) to create clap injected audio stream335and clap injected video stream340(seeFIG. 7and corresponding text for further details). In turn, transmitting system300sends synchronized clap injected media stream325to receiving device350over computer network345where receiving device350uses digital clapperboard synchronizer360to resynchronize the two streams (seeFIG. 6and corresponding text for further details).

FIG. 6is an exemplary diagram that depicts a digital clapperboard synchronizer using injected digital claps to synchronize an unsynchronized audio stream with a video stream. Receiving device350uses digital clapperboard synchronizer360to synchronize unsynchronized media stream355. Digital clapperboard synchronizer360uses metadata extractor600to extract audio clap information and video clap information from metadata header330. Metadata extractor600then loads the audio clap information (tone values) into preselected audio tones610and loads the video clap information into preselected colors660and preselected pixel coordinates670. Metadata extractor600also loads the extracted frame interval included in metadata header330into preselected frame interval640, which frame counter650uses to determine whether audio or video delays are beyond the frame intervals (seeFIG. 8and corresponding text for further details).

Digital clapperboard synchronizer360loads preselected audio tones610into audio clap detector620, which audio clap detector620uses to determine whether the inaudible tones (audio claps) are present in a given audio frame of clap injected audio stream335. If necessary, as discussed in detail below, digital clapperboard synchronizer360uses audio delayer630to delay clap injected audio stream335when clap injected audio stream335is determined to be ahead of clap injected video stream340(seeFIG. 8and corresponding text for further details).

Digital clapperboard synchronizer360also loads preselected colors660and preselected pixel coordinates670into video clap detector680, which video clap detector680uses to determine whether the preselected colors are present in the preselected coordinates in a given video frame. If necessary, as discussed in detail below, digital clapperboard synchronizer360uses video delayer690to delay clap injected video stream340when clap injected video stream340is determined to be ahead of clap injected audio stream335(seeFIG. 8and corresponding text for further details).

FIG. 7is an exemplary flowchart showing steps taken to inject digital claps into a media stream.FIG. 7processing commences at700whereupon, at step710, the process selects a random set of inaudible tones for audio claps. For example, the process may use a random number generator to select four values from 1-20 and 20,000-22,000. At step720, the process selects a random set of colors and a random set of pixel coordinates for video claps. For example, the process may use the random number generator to select values between 0-100 for red, green blue (RGB) values as well as x/y pixel coordinates between 0-1000. At step730, the process defines a frame interval to synchronously inject audio claps and video claps, such as every 500 frames.

At step740, the process creates a metadata header that includes the inaudible tone values, random color values, random coordinate values, and frame interval (seeFIG. 5and corresponding text for further details). At step750, the process selects the first clapperboard set, which includes an audio tone, a pixel color, and pixel coordinates. At step760, the process traverses through media stream305(audio/video) to the first frame interval. At step770, the process inserts the selected audio clap (audio tone) into the audio stream and inserts the selected video clap (color in pixel locations) into the video stream. At step780, the process sends the synchronized clap injected media stream with metadata header to client if the process is performing real-time streaming, or stores the clap injected media stream with the metadata header into a storage area if the process is performing non-real time streaming.

The process determines as to whether to inject more audio claps and video claps into media stream305(decision790). If the process should inject more audio claps and video claps into media stream305, then decision790branches to the ‘yes’ branch which loops back to select the next clapperboard set and inject audio claps and video claps at the predefined frame interval. This looping continues until the process should terminate (e.g., end of media stream305), at which point decision790branches to the ‘no’ branch exiting the loop.FIG. 7processing thereafter ends at795.

FIG. 8is an exemplary flowchart showing steps taken to synchronize an audio stream with a video stream based on injected digital claps.FIG. 8processing commences at800whereupon, at step802, the process captures metadata header330and extracts the preselected audio tones, colors, pixel coordinates, and frame interval information as discussed herein.

At step804, the process selects the first audio/video frame, and determines as to whether the selected audio frame includes audio clap information corresponding to the first clapperboard set (decision806). For example, the process checks whether the audio frame includes each of the four preselected inaudible tones. If the selected audio frame does not include the audio clap information, then decision806branches to the ‘no’ branch, whereupon the process determines as to whether the selected video frame includes the video clap information corresponding to the first clapperboard set (decision808).

If the selected video frame does not include the video clap information, then decision808branches to the ‘no’ branch and bypassing steps810-822because neither of the selected frames includes clap information. On the other hand, if the selected video frame includes video clap information but the corresponding audio frame does not include the audio clap information (from decision808), then decision808branches to the ‘yes’ branch whereupon, at step810, the process saves the expected audio clap information, saves the current video frame number, and pauses the video because the video stream is ahead of the audio stream at this point.

At step812, the process selects the next audio frame and determines as to whether the selected audio frame includes the expected audio clap information (decision814). If the selected audio frame does not include the expected audio clap information, then decision814branches to the ‘no’ branch, which loops back to select the next audio frame. This looping continues until the selected audio frame includes the expected audio clap information, at which point decision814branches to the ‘yes’ branch exiting the loop.

The process determines as to whether the selected audio frame is beyond the frame interval relative to the stored video frame number (decision816), such as greater than 500 frames from the stored video frame number. If the selected audio frame is beyond the frame interval, then decision816branches to the ‘yes’ branch whereupon, at step818, the process invokes an Error Message andFIG. 8processing thereafter ends at820because, for example, the audio stream and the video stream are unsynchronized more than the predefined frame interval.

On the other hand, if the selected audio frame is not beyond the frame interval, then decision816branches to the ‘no’ branch. At step822, the process unpauses the video stream and the audio stream and video stream are now synchronized.

Referring back to decision806, if the selected audio frame includes audio clap information, then decision806branches to the ‘yes’ branch whereupon the process determines as to whether the selected video frame includes the corresponding video clap information (decision824). If the selected video frame includes the corresponding video clap information, then decision824branches to the ‘yes’ branch, bypassing steps824-836and indicating that the audio stream and video stream are synchronized because they both include their respective clap information.

On the other hand, if the selected video frame does not include video clap information, then decision824branches to the ‘no’ branch whereupon, at step826, the process saves the expected video clap information, saves the audio frame number, and pauses the audio stream because the audio stream is ahead of the video stream at this point.

At step828, the process selects the next video frame and determines as to whether the selected video frame includes the expected video clap information (decision830). If the selected video frame does not include the expected video clap information, then decision830branches to the ‘no’ branch which loops back to select and process the next video frame. This looping continues until the selected video frame includes the expected video clap information, at which point decision830branches to the ‘yes’ branch exiting the loop.

The process determines as to whether the selected video frame is beyond the frame interval (decision832). If the selected video frame is beyond the frame interval, then decision832branches to the ‘yes’ branch whereupon, at step834, the process invokes and Error Message andFIG. 8processing thereafter ends at836because, for example, the video stream and the audio stream are unsynchronized more than the predefined frame interval.

On the other hand, if the selected video frame is not beyond the frame interval, then decision832branches to the ‘no’ branch. At step838, the process unpauses the audio stream and is now synchronized with the video stream.

The process determines as to whether to continue processing the receiving media stream (decision840). If the process should continue, then decision840branches to the ‘yes’ branch which loops back to select and process the next audio/video frame. This looping continues until the process should terminate, at which point decision840branches to the ‘no’ branch exiting the loop.FIG. 8processing thereafter ends at850.

FIG. 9is an exemplary diagram that depicts pseudo-code of a digital clapperboard injector. Pseudo-code900includes frame interval910, audio clap information920, and video clap information930. Frame interval910is set to 500 frames and audio clap information920is set to tones 10, 15, 20000, and 21000. Video clap information930includes four sets of video clap information, each of which includes color values and pixel coordinate values.

FIG. 10is an exemplary diagram that depicts pseudo-code of a digital clapperboard synchronizer. Receiving device350uses pseudo-code1000to perform steps shown inFIG. 8that evaluates clap injected audio stream335against clap injected video stream340.

When clap injected audio stream335is ahead of clap injected video stream340, pseudo-code1000pauses clap injected audio stream335(1010) until clap injected video stream340catches up, at which point pseudo-code1000unpauses clap injected audio stream335(1030).

Likewise, when clap injected audio stream335is behind clap injected video stream340, pseudo-code1000pauses clap injected video stream340(1020) until clap injected audio stream335catches up, at which point pseudo-code1000unpauses clap injected video stream340(1040).