Systems and methods for performing content aware video editing

A method implemented in a video editing device comprises retrieving media content and generating a user interface comprising a graphical representation of the retrieved media content on a first timeline component. The method further comprises analyzing the retrieved media content to extract attributes associated with the media content and generating a second timeline component in the user interface. At least a portion of the extracted attributes is arranged along the second timeline component with respect to time, and each of the portion of extracted attributes is represented by a corresponding graphical representation. Furthermore, each attribute corresponds to a segment in the media content. The method further comprises retrieving, based on the displayed attributes arranged along the graphical timeline component, a selection of at least one segment of the media content.

BACKGROUND

Over the years, digital video content has gained increasing popularity with consumers. With the ever-growing amount of audio and video content available to consumers through the Internet using computers, smart phones, and other sources, consumers have access to a vast amount of content and programming. Furthermore, many devices (e.g., PCs, DVD recorders) and services that are readily available allow consumers to record, time-shift or view on-demand video and audio content.

The availability of multimedia content in a vast array of digital formats has facilitated distribution of multimedia content because of the high degree of portability. A user may capture video of an event such as a graduation event, a wedding, or a performance. Such user-generated content can be lengthy and many times, contain such defects as poor lighting and shaking caused by movement of the camera while capturing the video. Off-the-shelf video editing applications provide users with the capability to incorporate special effects into captured images, audio and video. Some video editing/playback applications allow users to incorporate comments and tags at specific points within the video. However, video editing can be a complex and time-consuming task, particularly for users with less experience.

SUMMARY

Briefly described, one embodiment, among others, is a method implemented in a video editing device that comprises retrieving media content and generating a user interface comprising a graphical representation of the retrieved media content on a first timeline component. The method further comprises analyzing the retrieved media content to extract attributes associated with the media content and generating a second timeline component in the user interface. At least a portion of the extracted attributes is arranged along the second timeline component with respect to time, and each of the portion of extracted attributes is represented by a corresponding graphical representation. Furthermore, each attribute corresponds to a segment in the media content. The method further comprises retrieving, based on the displayed attributes arranged along the graphical timeline component, a selection of at least one segment of the media content.

Another embodiment is a video editing system that comprises a media interface configured to obtain media content and a content analyzer configured to analyze the media content and extract attributes associated with the media content, the attributes corresponding to defective segments in the media content. The system further comprises a user interface (UI) generator configured to generate a user interface comprising a graphical representation of the retrieved media content on a first timeline component. The UI generator is further configured to generate a second timeline component in the user interface, and at least a portion of the extracted attributes is arranged along the second timeline component with respect to time. Each of the portion of extracted attributes is represented by a corresponding graphical representation. The UI generator is further configured to retrieve, based on the displayed attributes arranged along the graphical timeline component, a selection corresponding to at least one segment of the media content.

Another embodiment is a non-transitory computer-readable medium embodying a program executable in a computing device. The program comprises code that generates a graphical representation of media content on a first timeline component and code that extracts attributes associated with the media content, the attributes corresponding to possible defects in the media content, and wherein each attribute corresponds to a segment in the media content. The program further comprise code that generates a user interface including a second timeline component, wherein at least a portion of the extracted attributes is arranged along the second timeline component with respect to time, wherein each of the portion of extracted attributes have a corresponding graphical component. The program further comprises code that retrieves, based on the displayed attributes arranged along the graphical timeline component, a selection comprising at least a portion of at least one segment of the media content.

DETAILED DESCRIPTION

One perceived shortcoming with traditional multimedia editing solutions is the degree of time and complexity involved in the editing of multimedia content such as video and audio clips. With many multimedia editing applications, users must first preview the entire video and manually identify points of interest before performing such editing operations as copying video segments of interest and creating new video segments by combining or re-ordering copied segments. Furthermore, depending on the quality of the video, the user may have to first identify segments within the video clip with such defects as poor lighting, poor contrast levels, the presence of artifacts, etc. before manually touching up the identified segment(s) to address the defects. As one will appreciate, this can be a time-consuming process, particularly with lengthy video clips.

Various embodiments are disclosed for providing users with an organized framework for editing video content based on analysis performed by a video editing system. In accordance with various embodiments, the video editing system receives multimedia content and analyzes the content to identify possible segments of interest. The video editing system also identifies possible defects within the content. The results of the analysis are then presented to the user in the form of a timeline-based user interface where attributes/characteristics of the content are displayed with respect to time. Such characteristics may include, for example and without limitation, zooming/panning motion by the camera, the identification of one or more faces, fast motion by objects within the content, and so on. Defects may include, for example and without limitation, segments with poor lighting, poor contrast levels, video shaking, and so on.

The presentation of a timeline-based user interface facilitates the selection of one or more segments by the user for editing purposes. For some embodiments, if the user selects a segment of interest that has an identified defect, the video editing system provides the user with an opportunity to rectify the defect. A modified version of the segment of interest is produced by the video editing system, thereby allowing the user to continue with the editing process on the selected segment. A description of a system for facilitating the video editing process is now described followed by a discussion of the operation of the components within the system.

FIG. 1is a block diagram of a video editing system102in which embodiments of the video processing techniques disclosed herein may be implemented. The video editing system102may be embodied, for example, as a desktop computer, computer workstation, laptop, a smartphone109, a tablet, or other computing platform and includes a display104and may include such input devices as a keyboard106and a mouse108.

For embodiments where the video editing system102is embodied as a smartphone109or tablet, the user may interface with the video editing system102via a touchscreen interface (not shown). In other embodiments, the video editing system102may be embodied as a video gaming console171, which includes a video game controller172for receiving user preferences. For such embodiments, the video gaming console171may be connected to a television (not shown) or other display.

The video editing system102is configured to retrieve, via the media interface112, digital media content115stored on a storage medium120such as, by way of example and without limitation, a compact disc (CD) or a universal serial bus (USB) flash drive, wherein the digital media content115may then be stored locally on a hard drive of the video editing system102. As one of ordinary skill will appreciate, the digital media content115may be encoded in any of a number of formats including, but not limited to, Motion Picture Experts Group (MPEG)-1, MPEG-2, MPEG-4, H.264, Third Generation Partnership Project (3GPP), 3GPP-2, Standard-Definition Video (SD-Video), High-Definition Video (HD-Video), Digital Versatile Disc (DVD) multimedia, Video Compact Disc (VCD) multimedia, High-Definition Digital Versatile Disc (HD-DVD) multimedia, Digital Television Video/High-definition Digital Television (DTV/HDTV) multimedia, Audio Video Interleave (AVI), Digital Video (DV), QuickTime (QT) file, Windows Media Video (WMV), Advanced System Format (ASF), Real Media (RM), Flash Media (FLV), an MPEG Audio Layer III (MP3), an MPEG Audio Layer II (MP2), Waveform Audio Format (WAV), Windows Media Audio (WMA), or any number of other digital formats.

As depicted inFIG. 1, the media interface112in the video editing system102may also be configured to retrieve digital media content115directly from a digital camera107where a cable111or some other interface may be used for coupling the digital camera107to the video editing system102. The video editing system102may support any one of a number of common computer interfaces, such as, but not limited to IEEE-1394 High Performance Serial Bus (Firewire), USB, a serial connection, and a parallel connection.

The digital camera107may also be coupled to the video editing system102over a wireless connection or other communication path. The video editing system102may be coupled to a network118such as, for example, the Internet, intranets, extranets, wide area networks (WANs), local area networks (LANs), wired networks, wireless networks, or other suitable networks, etc., or any combination of two or more such networks. Through the network118, the video editing system102may receive digital media content115from another computing system103. Alternatively, the video editing system102may access one or more video sharing websites134hosted on a server137via the network118to retrieve digital media content115.

The content analyzer114in the video editing system102is configured to analyze and identify attributes of the media content115retrieved by the media interface112in order to facilitate the selection of one or more segments within the media content115for editing purposes. Based on the attributes/characteristics identified by the content analyzer114, the user interface (UI) generator119generates a user interface that includes a graphical timeline component with the identified attributes arranged according to time.

The content analyzer114is further configured to identify possible defects within the media content115, where the identified defects are also presented to the user on the graphical timeline component. The interface presented to the user by the UI generator119allows the user to select defective segment(s) to modify (e.g., to increase the lighting or increase the contrast level). Based on the user input relating to the defective segment(s), the defects correction module116modifies or corrects the identified defect(s). The interface also allows the user to select one or more segments of interest based on the presented attributes on the timeline component for editing purposes.

FIG. 2is a schematic diagram of the video editing system102shown inFIG. 1. The video editing system102may be embodied in any one of a wide variety of wired and/or wireless computing devices, such as a desktop computer, portable computer, dedicated server computer, multiprocessor computing device, smartphone, tablet computing device, and so forth. As shown inFIG. 2, the video editing system102comprises memory214, a processing device202, a number of input/output interfaces204, a network interface206, a display104, a peripheral interface211, and mass storage226, wherein each of these devices are connected across a local data bus210.

The processing device202may include any custom made or commercially available processor, a central processing unit (CPU) or an auxiliary processor among several processors associated with the video editing system102, a semiconductor based microprocessor (in the form of a microchip), a macroprocessor, one or more application specific integrated circuits (ASICs), a plurality of suitably configured digital logic gates, and other well known electrical configurations comprising discrete elements both individually and in various combinations to coordinate the overall operation of the computing system.

The memory214can include any one of a combination of volatile memory elements (e.g., random-access memory (RAM, such as DRAM, and SRAM, etc.)) and nonvolatile memory elements (e.g., ROM, hard drive, tape, CDROM, etc.). The memory214typically comprises a native operating system217, one or more native applications, emulation systems, or emulated applications for any of a variety of operating systems and/or emulated hardware platforms, emulated operating systems, etc.

The applications may include application specific software which may comprise some or all the components (media interface112, content analyzer114, defects correction module116, UI generator119) of the video editing system102depicted inFIG. 2. In accordance with such embodiments, the components are stored in memory214and executed by the processing device202. One of ordinary skill in the art will appreciate that the memory214can, and typically will, comprise other components which have been omitted for purposes of brevity.

Input/output interfaces204provide any number of interfaces for the input and output of data. For example, where the video editing system102comprises a personal computer, these components may interface with one or more user input devices via the I/O interfaces204, where the user input devices may comprise a keyboard106(FIG. 1) or a mouse108(FIG. 1). The display104may comprise a computer monitor, a plasma screen for a PC, a liquid crystal display (LCD), a touchscreen display, or other display device.

In the context of this disclosure, a non-transitory computer-readable medium stores programs for use by or in connection with an instruction execution system, apparatus, or device. More specific examples of a computer-readable medium may include by way of example and without limitation: a portable computer diskette, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM, EEPROM, or Flash memory), and a portable compact disc read-only memory (CDROM) (optical).

With further reference toFIG. 2, network interface206comprises various components used to transmit and/or receive data over a network environment. For example, the network interface206may include a device that can communicate with both inputs and outputs, for instance, a modulator/demodulator (e.g., a modem), wireless (e.g., radio frequency (RF)) transceiver, a telephonic interface, a bridge, a router, network card, etc.). The video editing system102may communicate with one or more computing devices via the network interface206over the network118(FIG. 1). The video editing system102may further comprise mass storage226. The peripheral interface211supports various interfaces including, but not limited to IEEE-1294 High Performance Serial Bus (Firewire), USB, a serial connection, and a parallel connection.

Reference is made toFIG. 3, which is a flowchart300in accordance with one embodiment for facilitating video editing via content analysis. It is understood that the flowchart300ofFIG. 3provides merely an example of the many different types of functional arrangements that may be employed to implement the operation of the various components of the video editing system102(FIG. 1). As an alternative, the flowchart ofFIG. 3may be viewed as depicting an example of steps of a method implemented in the video editing system102according to one or more embodiments.

Beginning with block310, the UI generator119(FIG. 1) generates a user interface displayed to the user for selecting target media content for editing purposes. In accordance with some embodiments, the interface may include a first timeline component411(FIG. 4) to facilitate the selection of a segment for editing purposes. In block320, upon selection by the user of the target media content, the content analyzer114(FIG. 1) begins the content aware editing process by analyzing the selected media content115(FIG. 1) and identifying various attributes of the selected media content115.

In block330, the UI generator119generates another user interface that includes a (second) graphical timeline component431(FIG. 5), where the various attributes/characteristics (e.g., shaky video, poor lighting, the presence of zooming motion by the camera, the presence of one or more faces, fast/slow motion) identified by the content analyzer114are presented relative to time along the timeline component431of purposes of facilitating the selection of one or more segments within the target media content for editing purposes.

In block340, the UI generator119retrieves one or more selections from the user based on presentation of the timeline431component to the user. At decision block350, a determination is made on whether any of the selected segment(s) by the user contain defects identified by the content analyzer114. In block360, if a defective segment was selected, the user is prompted on whether to correct the identified defect.

Based on the user's response, the defects correction module116(FIG. 1) rectifies the defect(s) and generates a modified version of the selected segment. The method then proceeds to block370where the selected segments are shown in a workspace area of the user interface and in the first timeline, and the editing process is then performed on the selected segments. Returning back to decision block350, if none of the selected segments contains an identified defect, then the method proceeds to block370.

Reference is made toFIG. 4, which is an example user interface for editing a video clip in accordance with various embodiments. The example interface shown allows the user to perform such editing functions as copying and pasting of segments in order to create an edited video clip. Shown inFIG. 4is a master timeline411that corresponds to the edited video clip. In the example shown, the user has selected “segment1,” which then undergoes analysis to identify attributes and/or defects corresponding to the segment. To select additional segments using the content aware techniques disclosed, the user may invoke a selection menu407using a pointer401such as a mouse pointer. As described, the content aware techniques facilitate the selection of segments within a video by presenting different attributes/characteristics associated with the target media content in a timeline-based user interface, which the user utilizes for selecting segments.

Reference is made toFIG. 5A, which is an example user interface for facilitating the selection of segments of interest within a video based on the content aware techniques described. After selecting media content of interest (e.g., a video clip), the content analyzer114(FIG. 1) analyzes the entire video clip and identifies various attributes/characteristics associated with the video clip. For some embodiments, content analyzer114stores time stamp information associated with the presence of various attributes/characteristics at different points within the video. For example, the content analyzer114may determine that the camera zooms in and out for a duration of time spanning Time1to Time2.

To facilitate the selection of segments within the video, a timeline component431is provided in the user interface where the various attributes identified by the content analyzer114are arranged according to a time axis, where each attribute is represented by a distinct graphical component that corresponds to a segment. In the example ofFIG. 5A, for example, each graphical component comprises a bar. Furthermore, the width of each bar corresponds to the duration of the attribute/characteristic represented by the bar. For example, the timeline431conveys to the user that the camera performs a zooming motion where the occurrence of the zooming motion is represented by the bars410a,410bshown. As another example, the occurrence of a panning motion is represented by the bar414shown.

As another example, the timeline431shows three different bars416a,416b,416c, which correspond to three different instances in which one or more faces are shown in the video. The progression component405corresponds to the current point in time during playback of the content. Also shown are defects identified by the content analyzer114. As shown inFIG. 5A, the content analyzer114identifies a portion of the video in which the video exhibits poor contrast levels as represented by bar418. The timeline431further includes a bar420associated with a portion of the video in which the video exhibits shaky motion. Note that the attributes associated with the media content is not limited to attributes corresponding to the video portion and may include attributes corresponding to the audio portion as well. In the example shown, the timeline431further includes a bar422associated with a portion of the media content in which up-tempo audio is present. Other attributes corresponding to the audio portion of the media content may further include, for example, speech characteristics, noise levels, pitch, tonality, and energy level.

To select a segment of interest, the user may use a pointer401such as a mouse pointer on the interface to hover over the graphical representation corresponding to the target segment and click on the graphical representation of the segment to be selected. For purposes of this disclosure, a segment may comprise a video clip or a portion of a video clip. For implementations where the video editing system102is embodied as a smart phone109(FIG. 1) or tablet, the user may select segments of interest via a touchscreen display. For some embodiments, the user interface may also include a segmentation (not shown) for specifying locations relative to segments.

Reference is made toFIG. 5B, which illustrates a variation of the user interface ofFIG. 5A, where the user interface further comprises selection tools502,504for selecting and/or deselecting segments. In the example shown, the user utilizes the pointer401to first select a segment of interest (i.e., the segment represented by bar416a). The user then clicks on one of the components502,504to either add the selected segment to the “selected” workspace506or to the “deselected” workspace508. In the event that the user adds a segment to the “selected” workspace, the corresponding block of time is highlighted in the timeline component431, as shown. The “selected” workspace506and the “deselected” workspace are part of the video editing workspace.

FIG. 5Cillustrates yet another aspect of the selection process, where the user may manually select or define a subset517(i.e., “new segment”) of the segment of interest represented by bar414based on an overlapping portion relative to another segment represented by bar410a. The user may define the subset517using a pointer501or other segmentation tool. Significantly, this allows the user to define a segment that exhibits multiple attributes—in this case, the presence of both zooming and panning motions by the camera. In the example shown, the user could have alternatively defined another segment subset519that exhibits three attributes—the presence of both zooming and panning motions by the camera as well as the presence of faces.

FIG. 6illustrates the example user interface ofFIG. 5where the user is prompted in response to selection of a defective segment. As shown, the user selects a segment represented by bar418, which the content analyzer114(FIG. 1) determines as having poor contrast. In accordance with various embodiments, in response to selection by the user of a segment with an identified defect, the user is given the opportunity to remedy the defect.

In the example user interface, a dialog box602is presented to the user asking whether the user would like the defects correction module116(FIG. 1) to automatically remedy the defect—in this case, adjust the contrast levels. The user can either elect to automatically fix the defect or simply ignore the defect. As shown inFIG. 7, based on the user's response, the defects correction module116(FIG. 1) processes the defective segment702(which exhibits both poor contrast and shaky video) and generates a corrected version704, which is then selected for editing purposes. For example, with reference back toFIG. 5A, the defects correction module116may be configured to automatically fix defects associated with the segment represented by bar418and the segment represented by420based on the user's input. The user has full control over which defective segments undergo further processing by the defects correction module116.

Reference is made toFIG. 8, which is a flowchart800in accordance with an embodiment for editing video performed by the video editing system102ofFIG. 1. It is understood that the flowchart800ofFIG. 8provides merely an example of the many different types of functional arrangements that may be employed to implement the operation of the various components of the video editing system102(FIG. 1). As an alternative, the flowchart ofFIG. 8may be viewed as depicting an example of steps of a method implemented in the video editing system102according to one or more embodiments.

Although the flowchart ofFIG. 8shows a specific order of execution, it is understood that the order of execution may differ from that which is depicted. For example, the order of execution of two or more blocks may be scrambled relative to the order shown. Also, two or more blocks shown in succession inFIG. 8may be executed concurrently or with partial concurrence. It is understood that all such variations are within the scope of the present disclosure.

Beginning with block810, the media interface112(FIG. 1) in the video editing system102retrieves media content, and the UI generator119(FIG. 1) generates a user interface comprising a graphical representation of the retrieved media content on a first timeline component.

In block820, the content analyzer114(FIG. 1) analyzes the retrieved media content to extract attributes associated with the media content. In block830, the UI generator119generates a second timeline component in the user interface. At least a portion of the extracted attributes is arranged along the second timeline component with respect to time, and each of the portion of extracted attributes is represented by a corresponding graphical representation. Furthermore, each attribute corresponds to a segment in the media content. In block840, the UI generator119retrieves, based on the displayed attributes arranged along the graphical timeline component, a selection of at least one segment of the media content.