Web-based system for video editing

Disclosed are web-based systems and methods for editing digital videos. A graphical editing interface allows designating one or more videos to assemble into a video compilation. The graphical editing interface further allows specifying the portion of a constituent video of the video compilation that will be displayed when the video compilation is played. The graphical editing interface additionally allows the association of annotations—specifying, for example, slides, people, and highlights—with portions of the video. The associated annotations alter the appearance of the video compilation when it is played, such as by displaying slides, or text associated with the annotations, along with the video at times associated with the annotations. The associated annotations also enhance the interactivity of the video compilation, such as by allowing playback to begin at points of interest, such as portions of the video for which there is an associated annotation. The associated annotations can be created by selection of annotation tools of the graphical editing interface, where at least one of the annotation tools is created responsive to a user providing information associated with the tool.

TECHNICAL FIELD

The disclosed embodiments relate generally to web-based editing, composition, and annotation of digital videos.

BACKGROUND

Conventional web-based systems permitting the storage and display of digital videos typically allow video playback and have some rudimentary tools for supplementing or altering the original video. These tools typically oblige a user to manually specify aspects such as time ranges during which a particular condition applies. Conventional systems also lack mechanisms for selecting video clips from a larger video and compositing the videos or video clips into a single compilation.

SUMMARY

The present invention includes web-based systems and methods for editing digital videos. A graphical editing interface allows designating one or more videos to assemble into a video compilation. The graphical editing interface further allows specifying the portion of a constituent video of the video compilation that will be displayed when the video compilation is played. The graphical editing interface additionally allows the association of annotations—specifying, for example, slides, people, and highlights—with portions of the video. The associated annotations alter the appearance of the video compilation when it is played, such as by displaying slides, or text associated with the annotations, along with the video at times associated with the annotations. The associated annotations also enhance the interactivity of the video compilation, such as by allowing playback to begin at points of interest, such as portions of the video for which there is an associated annotation. The associated annotations can be created by selection of annotation tools of the graphical editing interface, where at least one of the annotation tools is created responsive to a user providing information associated with the tool.

DETAILED DESCRIPTION OF DRAWINGS

FIG. 1is a block diagram of a system architecture in accordance with one embodiment. As illustrated inFIG. 1, a video hosting server108includes a front end server124, a video server126, a network interface122, a video database128, and a user database140. Other conventional features, such as firewalls, load balancers, application servers, failover servers, site management tools, and so forth are not shown so as to more clearly illustrate the features of the system. Examples of a suitable video hosting server108for implementation of the system include the YouTube™ and Google Video™ websites; other video hosting sites are known as well, and can be adapted to operate according the teaching disclosed herein. It will be understood that the term “website” represents any system and method of providing content and is not intended to be limited to systems that support content provided via the Internet or the HTTP protocol. The various servers are conventionally implemented, whether as a single piece of software or hardware or as multiple pieces of software or hardware and can couple to the network105via the network interface122. In general, functions described in one embodiment as being performed on the server side can also be performed on the client side in other embodiments if appropriate.

A client130executes a browser132, and connects to the front end server124via a network105, which is typically the Internet, but may also be any network, including but not limited to a LAN, a MAN, a WAN, a mobile, wired or wireless network, a private network, or a virtual private network. While only a single client130and browser132are shown, it is understood that very large numbers (e.g., millions) of clients are supported and can be in communication with the video hosting server108at any time. The client130may include a variety of different computing devices. Examples of client devices130are personal computers, digital assistants, personal digital assistants, cellular phones, mobile phones, smart phones or laptop computers. As will be obvious to one of ordinary skill in the art, the present invention is not limited to the devices listed above.

In some embodiments, the browser132includes an embedded video player134such as, for example, the Flash™ player from Adobe Systems, Inc. or any other player adapted for the video file formats used in the video hosting video hosting server108. A user can access a video from the video hosting server108by browsing a catalog of videos, conducting searches on keywords, reviewing play lists from other users or the system administrator (e.g., collections of videos forming channels), or viewing videos associated with a particular user group (e.g., communities).

Video server126receives uploaded media content from content providers and allows content to be viewed by client130. Content may be uploaded to video server126via the Internet from a personal computer, through a cellular network from a telephone or PDA, or by other means for transferring data over network105known to those of ordinary skill in the art. Content may be downloaded from video server126in a similar manner; in one embodiment media content is provided as a file download to a client130; in an alternative embodiment, media content is streamed client130. The means by which media content is received by video server126need not match the means by which it is delivered to client130. For example, a content provider may upload a video via a browser on a personal computer, whereas client130may view that video as a stream sent to a PDA. Note also that video server126may itself serve as the content provider. Communications between the client130and video hosting server108, or between the other distinct units ofFIG. 1, may be encrypted or otherwise encoded.

Users of clients130can also search for videos based on keywords, tags or other metadata. These requests are received as queries by the front end server124and provided to the video server126, which is responsible for searching the video database128for videos that satisfy the user queries. The video server126supports searching on any fielded data for a video, including its title, description, tags, author, category and so forth.

Users of the clients130and browser132can upload content to the video hosting server108via network105. The uploaded content can include, for example, video, audio or a combination of video and audio. The uploaded content is processed and stored in the video database128. This processing can include format conversion (transcoding), compression, metadata tagging, and other data processing. An uploaded content file is associated with the uploading user, and so the user's account record is updated in the user database140as needed.

For purposes of convenience and the description of one embodiment, the uploaded content will be referred to a “videos”, “video files”, or “video items”, but no limitation on the types of content that can be uploaded are intended by this terminology. Each uploaded video is assigned a video identifier when it is processed.

The user database140is responsible for maintaining a record of all users viewing videos on the website. Each individual user is assigned a user ID (also referred to as a user identity). The user ID can be based on any identifying information, such as the user's IP address, user name, or the like. The user database may also contain information about the reputation of the user in the video context, as well as through other applications, such as the use of email or text messaging. The user database may further contain information about membership in user groups. The user database may further contain, for a given user, a list of identities of other users who are considered friends of the user. (The term “list”, as used herein for concepts such as lists of authorized users, URL lists, and the like, refers broadly to a set of elements, where the elements may or may not be ordered.)

The video database128is used to store the received videos. The video database128stores video content and associated metadata, provided by their respective content owners. The video files have metadata associated with each file such as a video ID, artist, video title, label, genre, and time length.

A video editing server150provides the ability to create compilations from, and add annotation to, videos in the video database128. The video editing server150has a user database152that maintains a record of all users using the video editing system. Each individual user is assigned a user ID (also referred to as a user identity). The user ID can be based on any identifying information, such as the user's IP address, user name, or the like. The user database152may also contain information about the reputation of the user in the video context. The user database may further contain information about membership in user groups. The user database may further contain, for a given user, a list of identities of other users who are considered friends of the user. (The term “list”, as used herein for concepts such as lists of authorized users, URL lists, and the like, refers broadly to a set of elements, where the elements may or may not be ordered.) In an embodiment in which the video hosting server108and the video editing server150are implemented using the same server system, then the user database140and the user database152are implemented as a single database.

The video editing server150keeps a record of various user video editing actions, such as aggregating videos into a compilation, clipping videos to more restricted portions, annotating the videos with information about slides, people or events, adding popup visuals such as text boxes, and the like. It then stores these records within an editing database154in association with the user ID from the user database152. The video editing server150also provides to entities such as the client130or the video hosting server108, for a given video, records of editing actions stored within the editing database154for that video. In one embodiment, the video editing server150is on a separate physical server from the video hosting server108, although in other embodiments the annotation functionality is included within the video hosting server108. Video editing server150may be operated by the same entity that operates video hosting server108, or may be a service provided by a third party, e.g., for a fee.

The editing database154stores information on video compilations, which are ordered collections of videos treated as a single composite video, and may include a single video. For example, editing database154stores in association with, e.g., a unique compilation identifier, an identifier of the user who created of the compilation, a total number of videos within the compilation, an identifier of each video to be played as part of the compilation, as well as an indicator (such as a time range) of the portion of each video to be played and an indicator of the order of the video within the compilation (such as “1” to indicate the first video in the compilation ordering). The editing database also maintains an association between each annotation and the appropriate portion of the annotated video or video compilation. In one embodiment, for example, the editing database154stores an identifier of the annotation type (e.g., a text box, or an annotation corresponding to a person) along with any information associated with that type (e.g., a text caption, or an ID corresponding to the person), a time stamp(s) of the portion of the video or compilation to which the annotation applies (e.g., from time 01:05 to time 01:26), an identifier of the video which the annotation annotates, and an identifier of a user who submitted the annotation (e.g., the user ID from the user database152). Many other storage implementations for annotations would be equally possible to one of skill in the art.

FIG. 2illustrates an example user interface for specifying the videos to be included in the compilation, according to one embodiment. In region210, a number of thumbnail representations of videos are displayed, in response to the entry of a query for videos specified in search area220. Specifically, region210displays videos returned in response to the query “penguin” entered in search area220. The query may locate videos from various sources, such as the video database128. Alternatively, a video can be specified directly via an identifier, such as a URL of a video within the video database128. A selected video is then displayed, e.g. via live video playback, in video region250. Videos are then added to the compilation via, for example, the Add button230. Videos currently added to the compilation are displayed in the region240. Selection of the Done button260causes the process to proceed to the editing phase.

FIG. 3Aillustrates an example user interface300used during the editing phase, according to one embodiment. Videos in the video set305correspond to the videos selected via the interface ofFIG. 2. In the example ofFIG. 3A, the compilation includes three videos305A-C, which are depicted using thumbnails representations of their content. Each video in the video set305has a remove button306A and a duplicate button306B; selection of the former removes the video from the video set305, and selection of the latter creates a new, separate instance of the video and adds it to the video set305. Selection of “Add Videos” button325redisplays the user interface ofFIG. 2, which can be used to select additional videos. One of the videos—video305A in the example of FIG.3A—is visually distinguished, for example by use of a highlighted box or other characteristic to designate that it is the currently selected video, and is shown in playback area310. The controls in playback control area315both control playback and display playback position within the set of selected videos. The play-pause button318is used to start and stop playback of the video. The scene seek button319allows a user to seek to the next “interesting” portion of the current video (or subsequent videos in the set of videos). As described further below, interesting portions include those with which annotation has been associated. The video segment bar316displays the relative lengths of each of the videos in the video set305, with dividers316A and316B demarcating the boundaries of the videos. That is, dividers316A and316B divide the video segment bar316into three distinct portions, the length of each reflecting the relative lengths of the played portions of the videos305. The thumb marker317displays the current playback location, including both the video that is being played and the location within that video. For example, the position of the thumb marker317inFIG. 3Aindicates that the current playback location is near the beginning of the first of the three videos305.

Timeline bar320represents the video currently being played, and thus corresponds to one of the segments on the video segment bar316. The current time marker322indicates the current playback location within the current video, and thus corresponds to the thumb marker317. The timeline bar320shows the times of the video currently being played, including the total length. In the example ofFIG. 3A, the time interval labels listed on the timeline bar320indicate that the current video (the first video305A, in the depicted example) has a length of approximately 1 minute and 30 seconds. Drag bars321A and321B indicate start and end times, respectively, for playback of the video. Thus, a user can drag the start drag bar321A to some time after the beginning of the current video, and/or the end drag bar321B to some time before the actual end of the video, and subsequent playback of the video will then begin and end at corresponding times. For example, referring toFIG. 3D, the start drag bar321A has been dragged forward from time 0:00 to 0:08, and the end drag bar321B has been dragged from time 1:30 back to time 0:45. Thus, when the video compilation is played back, the first video will start being played at time 0:08 and end at time 0:45, and the second and third videos will start and end at times according to their own drag bars321, which are displayed when the videos have been made active. A video is made active either explicitly, such as by the user clicking on the corresponding thumbnail representation of the video305, or implicitly, such as by playback of the video completing one video and proceeding to the next in sequence in the compilation.

The user interface300ofFIG. 3Aadditionally provides the ability to associate slides and other types of annotation with the videos in the compilation. For example, annotation tools region340displays the available annotation tools within a tool selection region345, which in the example ofFIG. 3Aincludes slides, people, and highlights. Regarding the slides annotation, current slide region330ofFIG. 3Aindicates that no slides have yet been associated with the videos305. Selecting region330, or upload button341, such as by clicking on them with the mouse, leads to the display of the user interface ofFIG. 3B.

FIG. 3Bdepicts two ways of associating slides with the videos: by uploading a local file (in this case, a Portable Document Format (PDF) file) via selection of the upload button305B, which leads to a standard file selection dialog box for specifying the file, or by specifying a URL in text box310B. Although in the example ofFIG. 3Bthe local file should be in PDF format, and the URL should correspond to a presentation hosted by the SLIDESHARE presentation hosting system, these options are merely for purposes of example, and in other embodiments other formats (e.g., PowerPoint) or presentation hosting systems may equally be employed.

FIG. 3Cdepicts the user interface300after a presentation has been associated with the videos305. The “Slides” icon of tool selection region345is highlighted to indicate that the slides annotation tool is the tool currently selected, the current slide region330now displays the first slide of the presentation, and slide representation region342displays a scrollable list of thumbnail representations of the slides in the presentation. Clicking on a slide thumbnail tool from region342associates that slide with the currently displayed portion of the video.

For example,FIG. 3Dillustrates the user interface300after annotation has been associated with the videos305. In particular, associated slides region347displays thumbnails of the slides from the region342ofFIG. 3C. As more slides are added, the slide thumbnails within region347shrink so that all the slides are displayed at once. Thus, as more slides are added, the content of a given slide may become difficult to discern at smaller thumbnail sizes. In order to allow a user to view the contents of the slide thumbnails at a sufficiently large size to be viewable, the user interface300enlarges the thumbnails in response to the user indicating an interest in them, such as moving the mouse cursor over them. For example, inFIG. 3D, the user has moved the mouse cursor over the fourth slide (“History of the Penguin”), and the fourth slide (as well as its neighboring slides) has accordingly grown in size. Moving the mouse cursor away from the fourth slide will cause it (and its neighboring slides) to return to its normal smaller size.

FIG. 3Dalso depicts slide markers355(as illustrated, markers displaying rectangle icons) placed on the timeline320at timeline locations corresponding to the time within the video at which the slide was associated with the video. In one embodiment, a slide is associated with the portion of the video beginning with the time that the slide was associated, and ending with a time that the next slide was associated, or with the end of the video compilation (i.e., the end of the last video in the sequence of ordered videos in the compilation), if there is no next slide.

FIG. 3Dfurther depicts user interface region350for defining annotation tools and adding annotation relating to people in the video. Selecting the “Add a new person” button351allows a user to specify data describing a person, such as the person's name and description. Once defined, an annotation tool for the person is added to a list within the region350and displayed to the user. For example, tools352A and352B respectively correspond to the person “Peter Penguin”, described as a “Documentary Star”, and “The Announcer”, described as an actor with the BBC. Each of these tools352can be edited via its associated Edit button. Selecting a people tool's “Tag now” button creates an annotation for the person corresponding to that entry and associates it with the current time of the current video, including displaying a person tag at the corresponding locations on the timeline320. For example, the timeline320ofFIG. 3Ddisplays two person tags356, represented with icons depicting a person.

FIG. 3Eillustrates the user interface300ofFIG. 3Cafter the addition of annotation. In addition to the slide and people annotations depicted inFIG. 3D,FIG. 3Efurther depicts a user interface region355for adding annotation related to highlights in a video. For example, region355depicts two text field entries, “Peter slides” and “The penguins begin to fly!” corresponding to two previously specified highlights, as well as a current text field357for adding a new highlight. Selecting the button358creates a new highlight for the text within the current text field357, associating it with the current time of the current video of the compilation and creating a marker tag at the corresponding time on the video. For example,FIG. 3Eillustrates two highlight marker tags357, depicted with star icons, indicating the highlights356A and356B.

FIG. 3Efurther depicts a user interface for selectively viewing the video compilation so as to focus on particular people or events. Selecting the icons362toggles display of their associated menus. For example, selecting the “People” or “Highlights” icons within icon set362causes menus361A and361B to appear or disappear, respectively. Menus361display the people and highlights annotations, respectively, with which videos of the video compilation have been annotated. For example, menu361A contains an item for “Peter Penguin”, the only people annotation that has been added to the compilation. Each selection of the “Peter Penguin” item causes video playback to begin at the next video portion (following the position indicated by the current time marker322) previously associated with the corresponding people tag. Similarly, menu361B contains a list of each highlight with which the video compilation has been tagged, the selection of any of which will cause playback of the video compilation at the corresponding time.

FIG. 3Eadditionally depicts displaying an indication of a region to which an annotation corresponds before a user causes playback to begin at the portion of the video corresponding to that annotation. Specifically, indicator359is located on the video segment bar316, its horizontal start and end positions indicating start and end times of a portion of the video to which the annotation applies. The indicator359can appear in response to a variety of events, such as hovering the mouse pointer over a visual representation of the annotation, such as an entry for annotations in menus361, a slide displayed in associated slides region347, or one of the annotation tags on timeline320. Moving the mouse pointer away from the visual representation of the annotation then causes the indicator359to disappear.

FIG. 3Ealso depicts the addition of textual comments to the video compilation. For example, moving the mouse cursor over the video playing in the video playback region310causes a prompt365to be displayed at the location of the mouse cursor. Responding to the prompt, e.g. by clicking the mouse at the current location, causes display of a text box entry area allowing the user to comment on the current portion of the video. For example, resulting comment area366displays a comment “He's sliding!” entered by the author of the video.

Any of the annotations displayed on the timeline320—e.g., slide, people, and highlight tags—can be associated with different portions of the video compilation by dragging their tag icons355-357to a different location on the timeline. Similarly, the annotations may be removed entirely by dragging their icons off the timeline, selecting them and pressing the “Delete” key, or the like.

The resulting video compilation may then be saved, e.g. via the Save button370, and published to the Internet. Related information, such as compilation title, compilation description, compilation tags (e.g. for finding the compilation as part of a search), whether the compilation is private to the author or also available to the public, and whether users other than the author can provide comments, can also be specified as part of publishing, e.g. as depicted in the sample user interface ofFIG. 3F. The compilation can also be shared, such as with users of social networking sites, as depicted inFIG. 3G.

During video playback, either while the video is being edited, or during post-publication viewing, any previously-specified annotations are displayed and can be used to alter the flow of control of the video compilation. For example, any slides that were associated with the video compilation are displayed within the associated slides region347ofFIG. 3D, the current slide being visually distinguished from the others (e.g., with an orange highlight) and also displayed in the current slide region330. Selecting a slide thumbnail within the associated slides region347causes video playback to begin at the point in time associated with that slide. Likewise, selecting a people or highlights tag from menus361causes video playback to begin at the highlight, or at the next instance of the people tag, if any. Additionally, as the video plays, the text associated with any of the markup tags or comments is displayed along with the video at the times of the video at which the tags/comments were specified. For example, comments (such as the comment area366ofFIG. 3E), the name of a people tag entry (e.g. “Peter Penguin”), and the title of an event (e.g. “Peter slides”) are displayed along with the video, such as over top of the playing video, or adjacent to it, for some given time period, e.g. 3 seconds.

It is appreciated that the exact components and arrangement thereof, the order of operations, and other aspects of the above description are purely for purposes of example, and a wide variety of alternative component arrangements and operation orders would be equally possible to one of skill in the art. For example, the video editing server150ofFIG. 1could be part of the video hosting server108, a server on the same local network as the video hosting server, or on a remote network from the video hosting server. As another example, the authentication server170could be separate from the video hosting server, or could be a component thereof. Further, some clients130could be configured not to communicate with the annotation server150, or the video editing server150not to provide annotations to the client130, in which case the client130obtains un-annotated video (e.g. clients in each country or language can have their own set of annotations). In some embodiments, the client130can perform annotation offline, without a network connection to the annotation server150, and later synchronize the annotations with the video editing server150when a network connection becomes available.

FIG. 4illustrates the steps involved in defining and playing video compilations, according to one embodiment. First, videos are received410, e.g. after a user selects the videos to include in the compilation via the user interface ofFIG. 2. These received videos make up the video compilation. Ordered representations of the videos are then displayed420. For example, the thumbnail representations305ofFIG. 3Aconstitute one such ordered representation, the thumbnail representations being arranged in an order specified by a user, such as the order in which the videos were added using the user interface ofFIG. 2, or a new order resulting from the user dragging the thumbnail representations into a new order. A timeline, such as timeline320ofFIG. 3A, is then displayed430responsive to a selection of one of the representations305. The timeline comprises start and end drag bars321corresponding to start and end times of a display portion of the video to display during video playback. When the user changes the position of the start or end drag bars321, this alters440the start or end time of the display portion. For example, advancing the start drag bar321causes the video, when played, to begin at a later point. The result of the prior actions defines450a video compilation comprising the display portions of the videos specified by the start and end drag bars321, where the display portions are ordered according to the corresponding ordered representations of the plurality of videos, e.g. the thumbnail representations305depicted inFIG. 3A. The video compilation can then be played460, where the display portions of the videos are played in the order of their respective videos, and the non-display portions are omitted from playback.

FIG. 5illustrates the steps involved in annotating a digital video, according to one embodiment. First, a video is received510, e.g. after a user selects a video via the user interface ofFIG. 2. A graphical user interface is then displayed520, the graphical user interface including the video, a timeline associated with the video, and annotation tools. For example, the user interface300ofFIG. 3Ddisplays people annotation tools352A and352B. At least one of the annotation tools is created responsive to a user providing information associated with the tool, such as a user selecting the “Add a new person” button351ofFIG. 3Dand specifying the name and description of a new person, leading to the creation of tools such as352. Likewise, the slide thumbnail tools of region342ofFIG. 3Care created responsive to the user specifying a slide presentation (e.g., via the user interface ofFIG. 3B) from which to create the slide thumbnail tools. An annotation is then associated530with a portion of the video responsive to selection by the user of one of the annotation tools. For example, if a user selects a slide annotation tool, a corresponding slide annotation is created and associated with a portion of the video starting at the current playback time of the video and ending at the beginning of the next slide annotation, or at the end of the video or video compilation, if there is no such next slide annotation. A visual indication of the annotation is additionally displayed on the timeline320at a location corresponding to the first portion of the video. For example,FIG. 3Edepicts a visual indication in the form of highlight annotation tag357added on the timeline320at a location corresponding to the current time of the video, as indicated by the current time marker322. These various visual indications can further be moved along the timeline to associate their respective annotations with different portions of the video, dragged off the timeline to remove them, and the like.

It should be noted that the process steps and instructions of the present invention can be embodied in software, firmware or hardware, and when embodied in software, can be downloaded to reside on and be operated from different platforms used by a variety of operating systems.

While the invention has been particularly shown and described with reference to a preferred embodiment and several alternate embodiments, it will be understood by persons skilled in the relevant art that various changes in form and details can be made therein without departing from the spirit and scope of the invention.