Method and system for annotating video content

Disclosed is a method and system for annotating video. A software application operating on a user computing device monitors a first display of video content transmitted over a network from a video server. The software application detects an action performed by a user of the user computing device on a first frame of the video content during display of the first frame. The action corresponds to the user identifying a portion of the displayed video frame upon which to overlay a visible marking on the first frame. The marking is associated with an object in the first frame. The software application iteratively derives a further position for the marking in a further frame of the video content, the further position associated with the location of the object in the further frame. The software application generates metadata associated with the user and transmits the metadata to a metadata sever.

FIELD

The present disclosure relates to video and, more specifically, to using a user computing device to annotate video content.

BACKGROUND

Software programs that enable users to annotate video content currently exist. For example, a user may store video on his or her computer. When the user displays the video, the user may use one of these annotation programs to annotate, or overlay markings on, the video. Typically, the user will pause the video and overlay one or more markings on the video. This annotated video screen can then, for example, be displayed or broadcasted. For example, sportscasters often pause a sports program such as a football game being broadcast on television and mark up the screen. The sportscaster may then allow the video clip to run with the markings being displayed. The sportscaster may do this to illustrate to their audience one or more points related to the football game.

SUMMARY

The present disclosure relates to the annotation of video content. A software application operating on a user computing device monitors a first display of video content transmitted over a network from a video server. The software application detects an action performed by a user of the user computing device on a first frame of the video content during display of the first frame of the video content. The action corresponds to the user identifying a portion of the displayed video frame upon which to overlay a visible marking on the first frame of the video content. The marking is associated with an object appearing at a location in the first frame of the video content. The software application iteratively derives a further position for the marking in a further frame of the video content, the further position for the marking associated with the location of the object in the further frame. The software application generates, during the display of the video content, video content metadata associated with the user. In one embodiment, the video content metadata includes coordinate and/or time information. The software application transmits the video content metadata to a metadata server for storage.

Upon receiving a request from the user for a subsequent display of the video content, the software application retrieves (a) the video content over the network from the video server and (b) the video content metadata from the metadata server in order to display the video content with the visible marking in the subsequent display of the video content in accordance with the video content metadata.

In one embodiment, the detecting step further comprises applying a Scale Invariant Feature Transform (SIFT) detector to a region of interest in the first frame to extract interesting points from the region. A SIFT vector is generated from the extracted points. The SIFT detector can be applied to the further frame to extract interesting points from the further frame. SIFT vectors are generated from the interesting points extracted from the further frame. The SIFT vector corresponding to the region of interest is compared with the SIFT vectors corresponding to the further frame to derive a new position of the interesting points to determine a new position of the object in the further frame.

The visible marking can be on or can surround the object. The detecting step can include applying an affine invariant point detector, a segmentation technique, a background modeling technique, and/or a supervised classifier technique.

These and other aspects and embodiments will be apparent to those of ordinary skill in the art by reference to the following detailed description and the accompanying drawings.

DESCRIPTION OF EMBODIMENTS

Embodiments are now discussed in more detail referring to the drawings that accompany the present application. In the accompanying drawings, like and/or corresponding elements are referred to by like reference numbers.

FIG. 1shows a block diagram of an embodiment of a system having a user computing device105in communication with a web server110, a video server115, and a metadata server120over a network130. The web server110transmits a software application140to the user computing device105. In one embodiment, the user of the user computing device105uses a browser executing on the user computing device105to navigate to a particular web site that is provided by the web server110. The user can then download the software application140over the network130to the user computing device105. In one embodiment and as described in more detail below, the software application140is a software module (e.g., a Flash module) that can render video content. The software application140can also be a software module that works together with a media player to display and annotate the video content.

After downloading the software application140, the user of the user computing device105retrieves video content145from the video server115. In one embodiment, the user uses the user computing device105to navigate to a particular web page, such as YouTube®, provided by the video server115. The user selects video content from the web page and the video server115transmits the selected video content145to the user computing device105. The software application140displays the video content145on the user computing device's display. As described in more detail below, the software application140enables a user of the user computing device105to overlay one or more visible markings on the video content. For example, the user can use a mouse or stylus associated with the user computing device105to overlay the markings on the video content. The visible markings can include, for example, lines and/or shapes (e.g., circles, squares, triangles, and/or comic-like balloons). The visible marking(s) are overlaid on or near an object in the video content. In one embodiment, the user overlays the visible marking(s) on the video content as the video content is playing. In another embodiment, the user overlays the marking(s) on the video content after pausing the playing of the video content. In particular, the user overlays visible markings on a first frame of the video content. When a further frame of the video content is displayed on the user computing device105, the software application140repositions the visible marking(s) on the further frame so that the marking(s) are still on or near their respective objects.

In one embodiment, the user selects a time segment during which the marking(s) are present in the video content. In this embodiment, when the time segment ends, the markings are then removed from the video content.

In one embodiment, the software application generates video content metadata150during the display of the video content. The video content metadata150is associated with the user. In one embodiment, the video content metadata150includes coordinate and time information. The amount of information generated can vary depending on the video content145itself and/or the marking(s). For example, the amount of video content metadata150generated can be different for marking(s) that are circles (e.g., x, y, radius) relative to video content metadata150associated with comic-like balloons (e.g., x,y, text) or squares (e.g., x1, y1, x2, y2). The software application140then transmits the video content metadata150to the metadata server120for storage.

In one embodiment, when the user computing device105receives a request from the user for a subsequent display of the video content145, the user computing device105retrieves the video content145from the video server115(or the video server115transmits the video content145to the user computing device105) and the video content metadata150from the metadata server120. In one embodiment, the software application140uses the video content metadata150associated with the user to display the video content145with the previously denoted visible marking(s).

The video content metadata150is independent relative to the video content itself and is associated with the user that created the marking(s). For example, if a first user logs onto the user computing device105, or server115, or metadata server120, and downloads the video content145, the software application140can download video content metadata150related to this first user for the video content145. The user computing device105then displays the video content145with the first user's markings overlaid on the video content145. At a later point in time, if a second user that has created different marking metadata logs onto their user computing device105or one of the servers, and downloads the same video content145, the software application140can download video content metadata150related to this second user for the video content145. The second user's user computing device105, which may the same or different device as used by the first user, can then display the video content145with the second user's markings overlaid on the video content145. The particular act of logging in is for the purpose of the user identifying him or herself to a service that maintains a user data base or profile data or other user information that can link the user created metadata to a specific user and specific video content. In one embodiment the user data is maintained by the entity that provides the video marking software, or affiliates thereof.

For purposes of this disclosure, (See, e.g.,FIG. 6) a computer or computing device such as the user computing device105and/or server110,115,120includes a processor and memory for storing and executing program code, data and software which may also be tangibly stored or read from any type or variety of well known computer readable storage media such as magnetic or optical discs, by way of non-limiting example. Computers can be provided with operating systems that allow the execution of software applications in order to manipulate data. Personal computers, personal digital assistants (PDAs), wireless devices, cellular telephones, internet appliances, media players, home theater systems, servers, and media centers are several non-limiting examples of computing devices. The user computing device105and the servers110,115,120can, in one embodiment, also include a display, such as a screen or monitor.

FIG. 2is a flowchart showing an embodiment of the steps performed by the user computing device105associated with the display of video content145. The user computing device105receives video content145from the video server115for display (step205). The user computing device105(in one embodiment, the software application140) displays a first frame of the video content in step210. The software application140monitors the display of the video content and detects (step215) a user action performed on the first frame of the video content, where the user action corresponds to the user identifying a portion of the displayed video frame upon which to overlay a visible marking on the first frame of the video content. The marking is associated with (e.g., on, surrounding, or near) an object in the video frame, such as a person, a person's head, a person's body, a person's arms or legs, an animal, a table, or any other object (animate or inanimate). The software application140then identifies, in step220, a further frame of the video content to be displayed and derives (step225) a further position for the marking in the further frame. In one embodiment, the further position of the marking is derived so that the marking maintains the same relative position to the object as the marking had in the first frame.

In one embodiment, the software application140tracks the objects of the video content145from one frame to the next in order to derive the further position(s) of the marking(s). Example methods of object tracking can include point detectors (e.g., Scale Invariant Feature Transform (SIFT) and/or affine invariant point detector), segmentation (e.g., mean-shift, graph-cut, and/or active contours), background modeling (e.g., mixture of Gaussians, Eigenbackground, wall flower, and/or dynamic texture background), and/or supervised classifiers (e.g., support vector machines, neural networks, and/or adaptive boosting).

As a specific example, the software application140can use a SIFT detector to track an object in a video frame. Given a first frame and a region or area inside the first frame, the software application140can apply the SIFT detector to extract “interesting” points in the region, or points that are identifiable relative to the rest of the region (e.g., a point that has a difference in intensity (or a difference in color or any other characteristic) above a threshold compared to the intensity (or color or characteristic) of points surrounding the identified point). These points are then described and stored as a SIFT vector. When a further frame of the video content145is displayed, the software application140applies the same SIFT detector over the entire image to obtain a group of SIFT vectors describing the interesting points. The software application140then compares the SIFT vectors extracted from the first frame (corresponding to the region of interest) to the SIFT vectors extracted from the further frame (corresponding to the entire image). When similar vectors are found, a new position of the interesting points can be derived and, as a result, the new position of the object (and the marking) can be determined. Thus, the tracking of the object and the corresponding derivation of a new position of the marking(s) associated with the object occurs at the user computing device105.

For example, if a red quarter is in the top left portion of a video, and the rest of the video frame is black, the SIFT detector can detect the change in color between the red points and the black points. The algorithm therefore detects the red points as “interesting” points. In a later frame, if the red quarter has moved to the top right portion of the video, the SIFT detector can detect that the quarter has moved during the playing of the video because of the characteristics (in this case, color) of the points of the quarter relative to the rest of the frame.

In step226, the software application140displays the further frame with the marking in its further position. In step227, the software application140generates video content metadata during display of the video content. The video content metadata is associated with the user who has overlaid the video content with the markings. The software application140then determines if there are additional frames of the video content to be displayed (step230). If so, steps220-230are repeated. If there are no additional frames to be displayed, the software application140transmits the video content metadata150that was generated in step227to the metadata server120in step235. In another embodiment, the software application140transmits the video content metadata150to the metadata server120in step227after generating the metadata150(without waiting to determine if there are additional frames to be displayed).

FIG. 3is a flowchart showing an embodiment of the steps performed by the user computing device105after transmitting the video content metadata150to the metadata server120. The user computing device105receives a request from the user for a subsequent display of the video content145(step305). The user computing device105(the software application140) retrieves the video content145from the video server115(or from memory of the user computing device105if the video content145has been saved locally) and also retrieves the video content metadata150for this video content145and associated with this user from the metadata server120(step310). The software application140uses the video content metadata150to determine the position(s) of marking(s) that were previously incorporated into the video content. In step315, the software application140displays the video content with the visible marking(s) in the subsequent display of the video content.

For example, if a first user uses the user computing device105to perform steps205-235ofFIG. 2and then steps305-315ofFIG. 3, the user computing device105will display the video content in step315having the markings that the first user previously overlaid onto the video content in step215. At a later point in time, a second user uses the user computing device105to perform the same steps205-235ofFIG. 2and steps305-315ofFIG. 3. When the user computing device105displays the video content for the second user in step315, the video content will be displayed differently than the display of the video content for the first user because the second user has different markings overlaid onto the video content (and, therefore, different video content metadata for the video content). Thus, the video content metadata is stored independent of the video content and is associated with the video content as well as the particular user who overlaid markings on the video content.

FIG. 4depicts an exemplary user interface of an embodiment of a first frame405of video content with markings overlaid on the first frame405. In particular, the user of the user computing device105uses software application407to display the first frame405of the video content. The software application407includes software tools410to overlay markings on the video frame. The first frame405includes a first person415and a second person420. In one embodiment, the user uses the software tools410to overlay a first marking425(in this example, a circle) around the first person's head. The user can also use the software tools410to overlay a second marking430(in this example, a rectangle) around the second person420.

FIG. 5depicts an exemplary user interface of an embodiment of a further frame505of the video content with the markings overlaid on the further frame505. In this further frame505, the first person415and the second person420are both in different locations compared with their locations in the first frame405inFIG. 4. The software application407determines these new locations of the first person415and the second person420and derives a new position for the first marking425and a new position for the second marking430. The new positions for the first marking425and the second marking430are based on the new positions of the first person415and the second person420. Thus, in this embodiment the first marking425is still a circle around the first person's head and the second marking430is still a rectangle around the second person420.

Various implementations of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof. These various implementations can include one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device. Such computer programs (also known as programs, software, software applications or code) may include machine instructions for a programmable processor, and may be implemented in any form of programming language, including high-level procedural and/or object-oriented programming languages, and/or in assembly/machine languages. A computer program may be deployed in any form, including as a stand-alone program, or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may be deployed to be executed or interpreted on one computer or on multiple computers at one site, or distributed across multiple sites and interconnected by a communication network.

A high level block diagram of a computing device is shown inFIG. 6. Computing device602is an example of computing device105and/or one or more of the servers110,115,120ofFIG. 1and contains a processor604which controls the overall operation of computing device602by executing computer program instructions which define such operation. The computer program instructions can be tangibly stored in a storage media612(e.g., magnetic or optical disk or other computer readable medium now known or to become known) and loaded into memory media610or read directly from media610when execution of the computer program instructions is desired. Computing device602also includes one or more interfaces606for communicating with other devices (e.g., locally or via a network). Computing device602also includes input/output608that represent devices that allow for user interaction with the computing device602(e.g., display, keyboard, mouse, speakers, buttons, etc.).

One skilled in the art will recognize that an implementation of an actual computing device will contain other components as well, and thatFIG. 6is a high level representation of some of the components of such a computing device for illustrative purposes, which may be, for example, a personal computer, PDA, wireless device, internet appliance, cellular telephone, or such processor driven technology. In addition, the processing steps described herein can also be implemented using dedicated hardware, the circuitry of which is configured specifically for implementing such processing steps. Alternatively, the processing steps can be implemented using various combinations of hardware, firmware and software.