Intelligent video navigation techniques

Automatic replay or skip ahead functionality can be configured to intelligently navigate to a portion of a video a user desires to view. The context at which a user selects intelligent navigation can be analyzed to determine where to initiate automatic replay or skip ahead. The context for intelligent navigation can be based on scene or shot segmentation data, closed captioning, aggregate video navigation data from a community of users of shared demographic traits and/or interest, and/or other metadata. In the case of automatic replay, playback of a portion of a video can include enhancements for that portion, such as providing closed captioning, display at a decreased frame rate (“slow motion”), zooming in/out on a portion of the frames of a video segment, among other enhancements.

BACKGROUND

Applications such as video-on-demand, video-sharing, digital video broadcasting, massive open online courses (MOOCs) or distance education, among other uses of digital video, are becomingly increasingly popular. An advantage of digital video over analog video is the relative ease in which users can navigate digital videos. For example, a conventional approach for navigating a digital video is the use of a “scrubber” that enables a user to quickly “fast-forward” by moving the scrubber forward and to quickly “rewind” by moving the scrubber backward. Another conventional approach for navigating a digital video is to provide a “skip ahead” button that fast-forwards a video by a specified number of seconds and a “playback” or “replay” button that “rewinds” the video by a specified number of seconds. However, these techniques of using a scrubber or skipping forward or reversing backward by a specified number of seconds may not accurately reflect where in the digital video the user has intended to navigate.

DETAILED DESCRIPTION

Systems and methods in accordance with various embodiments of the present disclosure may overcome one or more of the aforementioned and other deficiencies experienced in conventional approaches for enabling a user to navigate a digital video. In some embodiments, a user can select to replay a portion of a video, and based on the context of the video, the video will automatically be navigated to the portion the user most likely intended to replay. For example, the video may be replayed at the beginning of a shot or a scene depending upon how much of the shot or the scene has already been played. As another example, the video may be replayed at the beginning of dialogue, such as the start of a monologue, narration, conversation between multiple characters, among other possibilities.

In various embodiments, a segment of a video that is replayed can automatically be enhanced to improve video playback. For instance, in certain embodiments where a segment of video to be replayed includes dialogue, closed captioning can be presented during that segment. In some embodiments where a section of video corresponds to an action scene or action shot, the section can be played back in “slow motion” or at a decreased frame rate. In at least some such embodiments, sound and/or dialogue can automatically be adjusted to account for the slow motion or decreased frame rate, such as by increasing the length of time in between gaps in sound and/or dialogue. In some embodiments, extrinsic data, such as cast and character images and information, biographical information, quotations, trivia, goofs, related offerings, scene information, and other extrinsic data may be presented during replay. In some embodiments, a replayed segment can automatically be zoomed in or zoomed out of a particular portion of one or more frames of the replayed segment.

In some embodiments, the context of a section of video that is replayed can be based on scene or shot segmentation metadata. That is, a video may be segmented according to scenes and/or shots, and this data can be used to determine where in the video to begin replay. In other embodiments, replay context may be dependent upon analysis of closed captioning associated with a video, speech-to-text translation of dialogue, or similar text-based approaches. In some embodiments, video replay context may be based on aggregated data of users who have previously viewed the video. For example, a threshold number of previous viewers may have requested playback of a particular segment of the video, such as due to that segment including unintelligible audio or due to that segment being especially compelling to the previous viewers. As another example, a threshold number of previous viewers may have turned up the volume at a particular segment of the video indicating that the segment incorporates inaudible audio. Such data can be collected, and when a current viewer requests to replay that particular segment, playback can be based on the aggregated data. In certain embodiments, some combination of these various approaches, among others, can be utilized to determine where to begin playback of a video.

In some embodiments, a user can select to skip a portion of a video, and the video will automatically be forwarded to a subsequent section of the video using similar techniques discussed above. For example, the user can automatically skip to the end of a scene or a shot or the conclusion of certain dialogue. In other embodiments, scenes or shots may be associated with certain metadata that can be used to automatically advance a video to a later segment. This metadata can include information classifying a shot or a scene as corresponding to opening credits or closing credits, information indicating that the shot or scene includes objectionable material (e.g., nudity, adult language, violence, etc.), information indicating an end point of dialogue, information indicating an end point of a song or score incorporated within the shot or scene, among other possibilities. In still other embodiments, a scene can be automatically advanced based on the collective behavior of viewers who have previously viewed the video. In yet still other embodiments, some combination of these approaches, among others, can be used to determine automatic video fast-forwarding.

FIGS. 1A and 1Billustrate a conventional approach for enabling a user to navigate a digital video. In the example situation100ofFIG. 1A, a video of President Abraham Lincoln is shown on display screen104of computing device102. At a certain sequence, President Lincoln is delivering the Gettysburg address, in particular, the words112, “and dedicated to the proposition that all men are created equal,” which is output via speaker106of the computing device. The application, operating on computing device102and playing the video of President Lincoln presenting the Gettysburg address, includes a conventional user interface element108that enables a user to navigate to a portion of the video 10 seconds back in time. User interface element108is sometimes referred to as a playback, replay, or skip back button. User interface element108is provided, for example, in the event that a user didn't understand or missed a portion of dialogue to enable the user to playback or replay the missed dialogue. As another example, a particular video sequence may be especially compelling to a user and the playback or reply button enables the user to re-watch the sequence.

In the example situation120ofFIG. 1B, a user (not shown) has clicked on user interface element108, causing a previous portion of the video 10 seconds back in time to be displayed on display screen104of computing device102. Upon clicking the playback or replay button108, speaker106outputs a portion of the Gettysburg address at the words122, “our fathers brought forth on this continent.” The conventional approach thus plays a previous portion of a video a static amount of time (i.e., 10 seconds) back in time. This static approach may not navigate to the portion of the video the user is actually interested in re-watching but instead, such as here, navigates to the middle of dialogue (i.e., the middle of the first sentence of the Gettysburg address). A user is likely to be more interested in playing back a sequence of the video at the beginning of a scene or shot, or at the beginning of the dialogue, which is oftentimes not a static amount of time back in time.

FIGS. 2A and 2Billustrate an example approach for enabling a user to navigate a digital video that can be used in accordance with an embodiment. In the example situation200ofFIG. 2A, a video of President Abraham Lincoln is presented on display screen204of computing device202. Although a portable computing device (e.g., a smart phone, tablet computer, or portable digital media player) is shown that can be held in a user's hands, it should be understood that other types of computing devices can utilize aspects of the various embodiments as should be apparent in light of the teachings and suggestions contained herein. These other types of computing devices can include, for example, desktop computers, notebook computers, video gaming consoles, televisions, television set top boxes, digital video disc (DVD) players, digital media players, network appliances, among others. At a particular sequence in time, President Lincoln is delivering the words212, “and dedicated to the proposition that all men are created equal,” of the Gettysburg address. This portion of the Gettysburg address is output to speaker206located on the front of the device and the on same surface as the display screen to output audio to subject matter facing the front of the device, such as a user (not shown) viewing the display screen. It should be understood that, while the components of the example device are shown to be on a “front” of the device, there can be similar or alterative components on the “sides” or “back” of the device as well (or instead). Further, directions such as “front,” “side,” and “back” are used for purposes of explanation and are not intended to require specific orientations unless otherwise stated. In some embodiments, a computing device may include more than one speaker on the front of the device and/or one or more speakers on the back (and/or sides) of the device.

The example situation200ofFIG. 2Ais similar to the example situation100ofFIG. 1A. However, in the example200ofFIG. 2A, the application operating on computing device202and playing the video of President Lincoln, includes a user interface element208that provides for “intelligent” navigation to a previous portion of the video based on the context of the current portion video being played. In this example, an “i” icon is used to indicate that user interface element208provides for intelligent playback or replay. In other embodiments, a dynamic amount of time can be displayed to indicate the previous portion of the video that will be played back upon selection of the intelligent replay button. For example, if the context of the current portion of the video indicates that replay should occur 8.9 seconds back in time, “8.9s” can be displayed in lieu of the “i” icon. Further, it will be appreciated that the intelligent replay button208is not necessarily always displayed during the presentation of a video. Instead, in various embodiments, a user interaction during the playing of the video is required to cause the intelligent replay button208to be displayed and to be selectable (e.g., pressing a remote control button, moving a virtual cursor over the video, touching or having a finger hovering over a touchscreen displaying the video, etc.)

As mentioned, in some embodiments, the context for automatic video navigation can be based on scene or shot segmentation data. Video segmentation generally involves the partitioning of a video into its constituent parts, such as scenes, shots, and frames. A scene comprises a series of consecutive shots grouped together because, for example, they are captured in the same location or they share thematic content. A shot can be a sequence of frames recorded contiguously and representing a continuous action in time or space. A shot can also be an unbroken sequence of frames captured by a single camera. A frame is a single still image of a video. For example, a 90 minute film shot at 24 frames per second will contain 129,600 frames. Approaches for segmenting a video are discussed in co-pending U.S. patent application Ser. No. 14/577,277, filed Dec. 14, 2015, and entitled “Video Segmentation Techniques,” which is hereby incorporated herein by reference. Once a video has been segmented according to scene or shot, the segmentation data can be used for determining how far backward (or forward) to navigate the video when a user selects to replay (or fast-forward) a particular section of the video. For instance, in at least some embodiments, pressing a replay (or skip ahead) button can result in a video being navigated to a beginning (or end) of a scene or a shot.

In other embodiments, the context for automatic video navigation can be based on the dialogue corresponding to a section of video. For example, clicking on a replay (or skip ahead) button by a user can cause a video to be navigated based on closed captioning cues. Closed captioning is typically embedded in a video (e.g., CEA-608, CEA-708, among others) or stored as a separate file (e.g., TTML, DFXP, SMPTE-TT, SCC, EBU-TT, EBU-STL (binary), WebVTT, among others). CEA-608, also sometimes referred to as Line 21, is the National Television System Committee (NTSC) analog television standard used in the United States and Canada. In CEA-608, captions are encoded directly into a hidden area of the video stream by broadcasting devices. CEA-708 is the Advanced Television Systems Committee (ATSC) digital television standard used in the United States and Canada. Timed Text Markup Language (TTML) is a markup language that provides for the synchronization of text and other media, such as audio or video. Distribution Format Exchange Profile (DFXP) is a particular implementation of TTML that defines when and how to display caption data. Society of Motion Picture and Television Engineers-Timed Text (SMPTE-TT) is an extension of DFXP that adds support for three extensions found in other captioning formats and informational items but not found in DFXP, #data, #image, and #information. SMPTE-TT is also the FCC Safe Harbor format. If a content provider produces captions in SMPTE-TT format, the provider has satisfied its obligations to provide captioning in an accessible format. Scenarist Closed Caption (SCC) format contains SMTPE timecodes with corresponding encoded caption data as a representation of CEA-608 data. EBU-TT is a strict subset of TTML, and supported by the European Broadcast Union (EBU). That is, all EBU-TT files are valid TTML documents but not all TTML documents are valid EBU-TT files. EBU-STL is a binary format used by the EBU. EBU-STL files are stored as separate .STL files. Synchronized Accessible Media Interchange (SAMI) is based on HTML. WebVTT is a proposed standard for HTML5 video closed captioning. In at least some embodiments, closed captioning data can be utilized for enabling automatic video navigation. For example, if a user selects to replay (or fast-forward) a first section of video, the closed captioning data may be analyzed to determine a section of the video marking the beginning (or end) of a monologue, narration, or conversation from which automatic navigation can be initiated.

In still other embodiments, aggregate data from a community of users can be utilized for enabling automatic video navigation. In some embodiments, the community of users may share a demographic trait, such as age, gender, geographic location, income bracket, among others, with the specific user. In other embodiments, the community of users may share common interests, such as viewing, purchasing, recommending, and/or rating similar products and/or consuming common media items (e.g., video, music, books, video games, apps, etc.) In still other embodiments, the community of users can be based on a combination of shared demographic traits and interests. Once a community of users has been identified, their interactions with videos can be monitored and utilized for automatic video navigation for a particular user. For example, if a threshold number of users request playback (or skip ahead) from a first section of video to a second earlier (or later) section of video, such data can be a good indication of where to initiate automatic video navigation because it is more likely that the particular user would also prefer to navigate from the first section to the earlier (or later) second section.

In yet still other embodiments, other data associated with a digital video can be utilized for automatic video navigation, such as audio data (e.g., background song, score, or other audio), data regarding actors appearing in a scene, other text data (e.g., subtitles, location text, etc.) or other metadata tags or associations (e.g., action scene, opening credits, closing credits, etc.) In some embodiments, a combination of these approaches can be used for automatic video navigation, such as using a weighted combination based on the context at which replay or skip ahead is selected; using a first approach based on a first context, using a second approach based on a second context, using a third approach based on a third context, etc.; using multiple approaches at once and selecting the approach associated with a highest level of confidence; using multiple approaches at once and selecting a default approach when no single approach meets a threshold level of confidence; using multiple approaches at once and selecting a mean, median, or mode; among other possibilities. Various approaches known to those of ordinary skill in the art for combining data can be utilized within the scope of the various embodiments.

FIG. 3illustrates an example approach for providing navigation for a digital video that can be used in accordance with an embodiment. In particular,FIG. 3depicts an approach for replaying a portion of a digital video utilizing the automatic navigation techniques discussed herein. In the example300ofFIG. 3, a 90-minute digital video is represented as a timeline302in increments of seconds from 0 to 5400 seconds. A user (not shown) views the video up to a point in time304, which can represent a frame of the digital video (e.g., 1/24ths of a second if the video is shot at 24 frames per second (fps)). At that point in time or frame304, the user desires to replay a portion of the video and requests playback by clicking an intelligent replay button. In this example, a replay window306is identified for determining where in the video to initiate playback. In an embodiment, the replay window comprises a period of time between a minimum replay threshold (e.g., 8 seconds or 192 frames) and a maximum replay threshold (e.g., 12 seconds or 288 frames). In various embodiments, the minimum replay threshold can be as short as 3 seconds and the maximum replay threshold can be as long as 5 minutes. In some such embodiments, holding down a replay button or clicking on the replay button multiple times can cause replay to be initiated from a point further back in time or a point based on coarser segmentation data (e.g., DVD chapter segmentation data). In some embodiments, only one of a minimum replay threshold or a maximum replay threshold can be utilized. In other embodiments, no thresholds are utilized and either all previous played portions of a video are analyzed to determine the point of playback or some other heuristic can be utilized for determining the point of playback (e.g., navigating back to the beginning of a scene or a shot, beginning of dialogue, beginning of a background song or score, etc.)

In this example, after the replay window has been identified, the replay window can be analyzed to determine a point of playback308from where replay of the video is initiated. As discussed elsewhere herein, the point of playback can be based on video segmentation data, closed captioning, cumulative data from a community of users, other metadata, or a combination thereof. For example, the point of playback308can represent the beginning of a scene or a shot corresponding to point or frame304, the beginning of dialogue according to closed captioning data, a playback point selected by a community of users, and/or the beginning of a musical score, among other possibilities. In some embodiments, automatic playback can be accompanied with an enhancement between frames308and304, such as the segment of video between points or frames308and304being played in slow-motion or at a decreased frame rate, closed captioning being added to the segment, extrinsic data being displayed during the segment, zooming in/zooming out to a particular portion of the frames of the segment, among other enhancements. Although example300ofFIG. 3illustrates automatic replay, it will be appreciated by one of ordinary skill in the art that similar techniques can be utilized for enabling automatic fast-forwarding.

FIG. 4illustrates an example approach for enhancing a digital video that can be used in accordance with an embodiment. In the example ofFIG. 4, a stationary camera (not shown) is utilized to capture video of a moving car when the car is centered within the field of view of the camera, as seen in frame402a, as the car moves outside the center of the field of view of the camera, as seen in frame404, and as the car moves outside the field of view of the camera, as seen in frame406. It will be appreciated that there additional frames between frames402aand404and404and406but only frames402a,404, and406are provided for illustrative purposes. At a point in time corresponding to frame406, a user (not shown) elects to playback a portion of the video and performs a user interaction to cause user interface element410, an automatic playback button, to be displayed and to be selectable. For example, the user may tap a touchscreen of a portable computing device displaying the video or use a finger to hover over the touchscreen, use a mouse or other input element to move a virtual cursor over a video application playing the video, or press a remote control button of a television set, digital media player, or other video display appliance, among other possibilities. In this example, the scene or shot corresponding to frames402a,404, and406has been annotated in metadata as an action scene. Selection of the automatic replay button408causes playback to be initiated from the beginning of the shot or scene, as seen in frame402b. In this example, the selection of the automatic replay button also causes the video segment to be played in slow-motion or at a decreased frame rate because the scene is characterized as an action scene. Thus, instead of viewing the car as it moves from the center of the field of view of the camera until is completely outside the field of view of the camera as seen in frames402a,404, and406, the car can be seen from the center of the field of view of the camera as it slowly goes outside, but not completely outside, the field of view of the camera, as seen in frames402b,410, and412.

In some embodiments, a user selection of intelligent playback automatically causes a portion of a video to be played in slow-motion or at a slower frame rate, such as when a scene or shot is characterized as an action scene or action shot. In other embodiments, users can manually cause a portion of a video to be played in slow-motion or at a slower frame rate, such as holding down the intelligent replay button or tapping the intelligent replay button multiple times. In some embodiments, a portion of a video can be replayed at different, slower frame rates. For example, holding down the intelligent replay button for 1 second will cause a portion of a video to be replayed at 2× the normal frame rate, holding the button down 2 seconds will cause the portion of the video to be replayed at 4× the normal rate, holding down the button 3 seconds will cause the portion of the video to be replayed at 16× the normal frame rate, etc. Further, holding down the intelligent replay button for an extended period of time can cause cycling of the different frame rates and releasing the button will result in playback at the last displayed frame rate. In other embodiments, double tapping can cause the portion of the video to be replayed at 2× the normal frame rate, triple tapping can cause the portion of the video to be replayed at 4× the normal frame rate, etc.

In at least some embodiments, audio data can also be modified to correspond to a decreased frame rate using time compression/expansion or time stretching. As known in the art, time stretching leaves the pitch of a signal intact while changing its speed or tempo. There are two primary time compression/expansion techniques—Phase Vocoder and Pitch Synchronized Overlap-Add (PSOLA). Phase Vocoder uses a Short Time Fourier Transform (STFT) to convert the audio signal to the Fourier representation. As the STFT returns the frequency domain representation of the audio signal at a fixed frequency grid, the actual frequencies of the partial bins can be found by converting the relative phase change between two STFT outputs to actual frequency changes. The timebase of the audio signal can be changed by calculating the frequency changes in the Fourier domain on a different time basis, and then an inverse STFT is computed to regain the time domain representation of the signal. PSOLA is based on a correct estimate of the fundamental frequency of the processed audio signal. In one implementation, the Short Time Average Magnitude Difference function is calculated to find the minimum value. The timebase is changed by copying the input to the output in an overlap-and-add manner while simultaneously incrementing the input pointer by the overlap-size minus a multiple of the fundamental period. This results in the input being traversed at a different speed than the original data while aligning the estimated fundamental period.

FIG. 5illustrates an example approach for enhancing a digital video that can be used in accordance with an embodiment. In particular, the example video segment500comprises a couple talking at a restaurant with a first camera capturing a profile view of both the man and the woman, as shown in frame502a; a second camera capturing a front view of the man as he speaks, as depicted in frame504; and a third camera capturing a front view of the woman as she speaks, as illustrated in frame506. Again, it should be understood that there can be additional frames between frames502aand504and504and506. At a particular point in time corresponding to frame508, a user (not shown) performs an interaction causing a user interface element508, an intelligent replay button, to be displayed and to be selectable. The user subsequently clicks on the intelligent replay button, causing a portion of the video to be replayed at the beginning of the frame502b. In this example, video segment or scene500includes dialogue. Therefore, upon replay, the scene500is augmented with closed captioning510.

In some embodiments, a user selection of intelligent playback automatically causes a portion of a video to be incorporated with closed captioning, such as when a scene or shot includes dialog. In other embodiments, users can manually cause a portion of a video to be played with closed captioning, such as holding down the intelligent replay button or tapping the intelligent replay button multiple times. In some embodiments, holding down the intelligent replay button or multiple clicks of the intelligent replay button can enable different modalities to be selected by a user. For example, in an embodiment, holding down the replay button can enable a user to select to review a scene in slow motion, review the scene with closed captioning, review the scene with extrinsic data, review a zoomed in/zoomed out perspective of the scene, among other possible enhancements.

Examples of the extrinsic data that can be presented may include names or descriptions of performers in a video, biographies or filmographies of the performers, commentary, trivia, mistakes, user comments, image data, and/or other data. The extrinsic data may include curated data that is professionally managed, verified, or is otherwise trustworthy, and/or non-editorially curated sources (e.g., “Wiki” sources). For example, the extrinsic data may include cast/crew data, quote/trivia data, soundtrack data, product data, and/or other data. The cast/crew data can include the name, biographical data, character information, images, and/or other data describing cast members who perform in a video or crew members who are involved in the production of the video. The biographical data may include various information such as stage name, birth name, date of birth, date of death, an editorially curated biography, and/or other information.

The quote/trivia data may include various quotations from characters, trivia items, goofs, and other interesting tidbits of information for the video and may be correlated with times of appearance in in the video and/or scenes of appearance in the video. The soundtrack data may include various information about the audio of the video. For example, the soundtrack data may identify that a particular audio track is being used at a certain time in the video or during a certain scene of the video. The soundtrack data may indicate whether the audio corresponds to a title or theme track. In addition, the soundtrack data may identify performers who vocally perform characters in the audio. Such performers may be considered cast members. However, such performers may differ from cast members who visually perform the same characters in some cases. One such case is where, for example, when a song is recorded by a vocalist and a different performer merely lip-syncs to the recorded song in the video.

The product data may identify associations of products with times or scenes in a video. The products may correspond to any item offered for purchase, download, rental, or other form of consumption. For example, a particular brand of potato chips may be shown and/or mentioned in dialogue of a movie. The product data may be used to promote products that are related to various scenes in the video at the appropriate times. Such promotions may be rendered relative to a position of the product within a frame of the video. Such products may also include books, electronic books, soundtrack albums, etc. that are related to the video. For example, the video may be an adaptation of a book, or the album might be for the soundtrack of the video.

The image data may correspond to images of a performer which are taken when the performer is not performing a particular character. For example, such an image might be taken at an awards ceremony, at a press conference, at an informal setting, and/or elsewhere. Such an image may be a headshot or other image. Multiple generic images may be provided for a particular performer. For example, a performer may have a lengthy career, and the performer's image data may be included for various times within the career.

Although the example ofFIG. 5illustrates automatic replay, it will be understood by one of ordinary skill that similar techniques can be implemented for automatic fast-forwarding.

FIG. 6illustrates an example process600for enabling intelligent video navigation that can be used in accordance with an embodiment. It should be understood that, for any process discussed herein, there can be additional, fewer, or alternative steps performed in similar or alternative orders, or in parallel, within the scope of the various embodiments unless otherwise stated. The process may begin by powering on a computing device and playing a video. The process600may continue by obtaining a request to replay a portion of the video as the video is being played602. For example, a user may cause an intelligent replay button to be displayed and to be selectable, such as by tapping on a touchscreen presenting the video or using a finger to hover over the touchscreen; clicking on the video as it is being played using an input element, such as a mouse, trackpad, or pointer stick; or pressing a remote control button as the video is being played. After the intelligent replay button is displayed, the user can select the intelligent replay button, such as by tapping or clicking on the button. A current frame of the video is then obtained604. Each previous frame of a plurality of previous frames is analyzed based on video segmentation data606to determine an identified frame from which to initiate playback. In some embodiments, the plurality of previous frames to be analyzed can be determined by setting a minimum replay threshold and a maximum replay threshold such that only those frames within this replay window are analyzed. In an embodiment, the minimum replay threshold can be set to 8 seconds and the maximum replay threshold can be set to 12 seconds. In other embodiments, the minimum replay threshold can be set as low as 3 seconds and the maximum replay threshold can be set as high as 5 minutes. In some such embodiments, holding down the replay button or clicking on the replay button multiple times may result in replay from a point further back in time or a point based on coarser segmentation data, such as DVD chapter data. In still other embodiments, only one of a minimum replay threshold or a maximum replay threshold can be set. For instance, in one embodiment, when a current frame corresponds to an action scene, a maximum threshold of fifteen minutes can be set and all of the frames between the current frame and the 21,600 previous frames can be analyzed to determine a beginning of scene or a shot corresponding to the current frame.

At a decision point608, it is determined whether one of the previous frames marks the beginning of a scene or a shot. If a previous frame does not mark the beginning of a scene or a shot, a next frame is analyzed. If a previous frame is determined to mark the start of a scene or a shot, that identified frame is selected as the playback point, and playback can be initiated from that frame610. During playback, the segment between the identified frame and the current frame can be enhanced612. For example, if the scene includes dialogue, playback can be enhanced with closed captioning. As another example, if the scene corresponds to an action scene, the scene can be played back in slow motion or at a decreased frame rate. In at least some embodiments, the audio data, including speech, can be time-stretched and aligned with the corresponding video data as discussed elsewhere herein. In some embodiments, the enhancement may be the display of extrinsic data, such as cast and character images and information, biographical information, quotations, trivia, goofs, related offerings, scene information, and other extrinsic data.

Although the example process600is directed towards intelligent replay, it will be appreciated by one of ordinary skill in the art that similar techniques can be utilized for intelligent fast-forwarding. For example, instead of analyzing a set of previous frames for determining the beginning of a scene or a shot, intelligent fast-forwarding can be implemented by analyzing a set of successive frames for determining the end of a scene or a shot. In at least some embodiments, intelligent fast-forwarding can also include time-compression of audio data and alignment with corresponding video data as discussed elsewhere herein.

FIG. 7illustrates an example process700for enabling intelligent video navigation that can be used in accordance with an embodiment. The process may begin by playing a video on a computing device. The process700may continue by receiving a request to playback a portion of the video while the video is being played702. A current frame, at the point that the playback request is made, is obtained704. Subsequently, each previous frame of a previous set of frames is analyzed based on closed captioning data706to determine a playback point. In particular, the previous set of frames is analyzed to determine whether a previous frame marks the beginning of dialogue708. As in the example process600ofFIG. 6, the previous set of frames analyzed can be based on a minimum replay threshold and/or a maximum replay threshold to reduce the amount of processing required of process700. If a previous frame is determined not to mark the beginning of dialogue, a next frame is analyzed. On the other hand, if a previous frame is determined to mark the beginning of dialogue, that identified frame is selected as the point of playback. The video is then replayed from the identified frame710. During playback, the portion of video between the identified frame and the current frame can be enhanced712, such as incorporating closed captioning, playing the segment at a decreased frame rate, or displaying extrinsic data, as discussed elsewhere herein. Although the example process700is directed towards intelligent replay, it will be understood by one of ordinary skill that a similar approach can be utilized for intelligent fast-forwarding. For instance, instead of analyzing a set of previous frames for determining the beginning of dialogue, intelligent fast-forwarding can be implemented by analyzing a set of successive frames for determining the end of dialogue.

As mentioned, some embodiments enable different functionality based on user interaction with the intelligent replay or fast-forward button. For example, holding down the intelligent replay button can bring up a menu enabling a user to select closed captioning enhancement or slow motion enhancement. As another example, tapping once on the intelligent replay button can cause a segment of video to be replayed at 2× the normal frame rate, tapping twice will cause the segment to be replayed at 4× the normal frame rate, tapping three times will cause the segment to be replayed at 8× the normal frame rate, etc.

The processes600ofFIG. 6 and 700ofFIG. 7are directed towards streaming video, and the intelligent replay functionality is provided as part of a streaming service. Other embodiments are directed towards digital video stored on a computing device, and the intelligent replay functionality is provided as part of a digital video playing application. In some embodiments, the digital video playing application may connect to a remote server to obtain data to provide intelligent video navigation functionality, such as video segmentation data or aggregate video navigation data from a community of users. In other embodiments, video segmentation data, aggregate video navigation data, and the like can be encoded in a stored digital video or stored as separate file but associated with the digital video, and the digital video playing application does not require network connectivity.

FIG. 8illustrates an example computing device800that can be used to perform approaches described in accordance with various embodiments. In this example, the device includes four cameras808located at the top and bottom on each of a same and opposite side of the device as a display element806, and enabling the device to capture images in accordance with various embodiments. The computing device also includes an inertial measurement unit (IMU)812, comprising a three-axis gyroscope, three-axis accelerometer, and magnetometer that can be used to detect the motion and/or orientation of the device.

FIG. 9illustrates a logical arrangement of a set of general components of an example computing device800. In this example, the device includes a processor902for executing instructions that can be stored in a memory component904. As would be apparent to one of ordinary skill in the art, the memory component can include many types of memory, data storage, or non-transitory computer-readable storage media, such as a first data storage for program instructions for execution by the processor902, a separate storage for images or data, a removable memory for sharing information with other devices, etc. The device typically will include some type of display element906, such as a touchscreen, electronic ink (e-ink), organic light emitting diode (OLED), liquid crystal display (LCD), etc., although devices such as portable media players might convey information via other means, such as through audio speakers. In at least some embodiments, the display screen provides for touch or swipe-based input using, for example, capacitive or resistive touch technology. The device in many embodiments will include one or more cameras or image sensors908for capturing image or video content. A camera can include, or be based at least in part upon any appropriate technology, such as a CCD or CMOS image sensor having a sufficient resolution, focal range, viewable area, to capture an image of the user when the user is operating the device. An image sensor can include a camera or infrared sensor that is able to image projected images or other objects in the vicinity of the device. Methods for capturing images or video using a camera with a computing device are well known in the art and will not be discussed herein in detail. It should be understood that image capture can be performed using a single image, multiple images, periodic imaging, continuous image capturing, image streaming, etc.

The device, in many embodiments, will include at least one audio element910, such as one or more audio speakers and/or microphones. The microphones may be used to facilitate voice-enabled functions, such as voice recognition, digital recording, etc. The audio speakers may perform audio output. In some embodiments, the audio speaker(s) may reside separately from the device. The device, as described above relating to many embodiments, may also include at least one or more motion and/or orientation elements912that provide information such as a position, direction, motion, or orientation of the device. These one or more motion and/or orientation determining elements912can include, for example, accelerometers, inertial sensors, electronic gyroscopes, electronic compasses, and GPS elements.

The computing device also includes various power components914known in the art for providing power to a computing device, which can include capacitive charging elements for use with a power pad or similar device. The computing device can include one or more communication elements or networking sub-systems916, such as a Wi-Fi, Bluetooth, RF, wired, or wireless communication system. The device in many embodiments can communicate with a network, such as the Internet, and may be able to communicate with other such devices. In some embodiments the device can include at least one additional input element918able to receive conventional input from a user. This conventional input can include, for example, a push button, touch pad, touchscreen, wheel, joystick, keyboard, mouse, keypad, or any other such component or element whereby a user can input a command to the device. In some embodiments, however, such a device might not include any buttons at all, and might be controlled only through a combination of visual and audio commands, such that a user can control the device without having to be in contact with the device.