Analysis of media content via extensible object

A system and method are provided for analyzing media content and generating related metadata as it is provided to a computer. In one embodiment, the system includes an extensible analysis object managing a plurality of plug-ins for analyzing the media content and generating metadata related to the media content.

BACKGROUND

Capturing video includes the process of transferring media content from a recording device such as a digital video camcorder to a computer. A user is required to capture video in order to edit the media content of the video with a computer non-linear video editing system, or to store the media content on a computer. Recording devices (e.g., digital video camcorders) may have any number of memory systems with a common one being a tape based memory system. Capturing from tape based recording devices is a real-time process such that capturing 1 hour of video to a computer requires approximately 1 hour of time. Other types of recording devices (e.g., flash based memory, optical drive based memory, or magnetic drive based memory) have similar requirements.

Video capture applications typically allow the user to preview the media content while it is being captured. For example, the media content is stored on a recording device in the form of a DV-AVI video clip or MPEG-2 video clip. In order for a user to preview the media content as it is streamed to the computer, video capture applications must decode the video clip from its native format into an uncompressed format as it is received.

Users are required to manually apply processes to generate metadata, correct and enhance the video after completing the capture of the media content to the computer. Additionally, the generated metadata is stored in a location defined by the process generating the metadata. Since metadata generating applications are made by a number of different developers, there exists no common location and format of metadata generated for any given video clip. Thus, it is difficult for an application produced by one developer to use or even access metadata generated by an application produced by another developer.

SUMMARY

Embodiments of the invention include a system for analyzing media content, generating metadata related to the media content, and for storing the media content and metadata in a media file. The metadata is generated by at least one analysis object of a plurality of analysis objects which analyzes the media content and stores metadata to a metadata queue. Additional analysis objects are added to the system by interfacing an application programming interface (API) of the system. Thus, the metadata is stored in a consistent format and location such that any application can access the metadata, additional analysis tools can be easily added to the system, and metadata is associated with the media content it relates to.

DETAILED DESCRIPTION

Referring now toFIG. 1, a system for capturing media content from a recording device to a computer according to one embodiment of the invention is shown. The recording device102provides media content such as video, audio, photographs, or a combination thereof to the computer104. The recording device102may provide the media content in any format by any method. For example, the recording device102may stream the media content to the computer104, or it may render the media content for the computer104. The recording device102may provide the media content via a wired connection or a wireless connection to the computer104, and it may provide the media content at any speed. Alternatively, a computing device may transmit data representative of the media content to the computer104. The media content may be in DV-AVI format, MPEG-2 format, or any other format. Regardless of how the media content is provided to the computer104and the format of the media content, the computer104stores the media content in the original format in which it is provided by the recording device102in a media content queue106.

A processor126of the computer104executes computer executable instructions for storing, moving, and analyzing the media content. The computer executable instructions are represented by software objects including a transcoder108, a preview generator110, an extensible analysis object112, plug-ins116and118, and a table of contents (TOC) object122. Memory objects include a media content queue106, a media file124and a metadata queue120for storing the media content in various forms or for storing data relating to the media content as determined by the processor126.

The media content queue106provides the stored media content to a transcoder108. The transcoder108decodes the media content from its original format into another format. In one embodiment of the invention, the transcoder108decodes the media content into media information. The transcoder108provides the media content to an extensible analysis object112and optionally to a preview generator110. The optional preview generator110generates a playback of the media content provided to the computer104by the recording device102on a user display114of the computer104as the media content is received to allow a user to view the media content being transferred to the computer104. An application programming interface (API)128permits the extensible analysis object112to interface one or more of a plurality of plug-ins as selected by the user or an application. The processor126executes a first selected plug-in116which receives the media content via the extensible analysis object112and API128and examines the media content for a first characteristic. For example, the first plug-in116may analyze the media content for any characteristic including an audio pattern, a video pattern, a face, a color histogram, a motion vector analysis, a date stamp, a timecode, a color set, a scene change, an object, or a person's voice. The first plug-in116then generates first metadata relating to the first characteristic and provides it to the extensible analysis object112according to the API128. If selected, a second analysis plug-in118receives the media content according to the API128and examines it for a second characteristic. The second plug-in118then generates second metadata relating to the second characteristic and provides it to the extensible analysis object112according to the API128. The extensible analysis object112stores the first and second metadata in a metadata queue120as it is received from the plug-ins. The metadata may be stored in any order in the metadata queue120. In an alternative embodiment of the invention, the plug-ins provide metadata directly to the metadata queue120. In an embodiment of the invention, the media content is provided directly to all analysis objects (i.e., to the plug-ins without the need for the extensible analysis object112) and the analysis objects store the metadata in the metadata queue120such that the extensible analysis object112may be eliminated.

In an embodiment of the invention, the transcoder108may provide the media content to analysis objects without decoding the media content. The analysis objects must either be capable of decoding the media content themselves, or capable of analyzing the media content in the format in which it is received.

In one embodiment of the invention, processor126executes a table of contents object122to generate a table of contents based on the metadata in the metadata queue120. The table of contents object122writes the metadata and the table of contents to the media file124. The media file124stores the table of contents near the beginning of the file, followed by the media content, and then the metadata. The table of contents indicates what metadata is in the file, and where it is located in the file. In an embodiment of the invention, the metadata is stored in the media file124without a table of contents such that the TOC object122is not necessary.

The media content stored in the media file124is provided by the transcoder108. After the transcoder108has decoded the media content in the media content queue106, it encodes the media content in a new format, or provides the media content in its original format to the media file124. The format may be selected by the user, or by preference of the system or computer104.

Providing the media content to the computer104, transcoding the media content, previewing the media content, and analyzing the media content may occur simultaneously. That is, at some point all of the processes may be executing at the same time even though analyzing the media content may take longer to complete than the other processes. As the first of the media content is provided to the computer104, transcoding begins with decoding the media content. The decoded media content is analyzed and reviewed while additional media content is being provided to the computer104and decoded by the transcoder108. The media content (e.g., media information) encoded by the transcoder108is stored in the media file124along with the metadata from the metadata queue120and along with the TOC generated by the TOC object122. Thus, all of the processes can occur simultaneously.

Referring now toFIG. 2, a method for capturing and storing media content in a computer according to one embodiment of the invention is illustrated. At202, media content is provided to the computer104. The media content may be provided from a recording device such as recording device102, or any other device having media content such as a digital camera, another computer, or an audio recording device. At204, the computer104stores the provided media content in a media content queue106in the native format in which it is received. This minimizes loss of the data provided to the computer104and enables performing simultaneous processes on the media content that may not be able to operate in real time. The media content is decoded from its original format to an uncompressed format by a transcoder108at206, and an optional preview of the decoded content may be provided to a user on the user display114at218.

At208, the decoded media content from the queue106is analyzed for characteristics. Such characteristics may include an audio pattern, a video pattern, a face, a date stamp, a timecode, a color set, a scene change, an object, a color histogram, a motion vector analysis, and a person's voice. The analysis can be conducted by independent analysis objects, or by an extensible analysis object112which invokes one or more plug-ins116and/or118to examine the media content and generate metadata related to the media content. An example of analyzing media content for an audio pattern is analyzing a video clip for the song “Happy Birthday.” If the song is detected, metadata is generated indicating that the clip includes someone's birthday. An example of analyzing media content for a video pattern is analyzing a video clip for a ball passing through a hoop which can indicate that the video was taken at a basketball game. Analyzing a video clip for a face or a person's voice allows a user to search for video clips with a particular person in them. A date stamp or timecode allow a user to organize video clips chronologically. Analyzing video for a color set or an object can tell a user that a video clip where a video clip was taken. For example, if a scene is dominated by red and green, or if a pine tree and ornaments are present, then the video probably relates to Christmas. Additional types of analysis can be added by installing additional plug-ins. The user may specify that all available types of analysis may be performed, or only some of the available analysis is to be performed. Each analysis would generate metadata indicative of the analysis. At210, the metadata generated during analysis of the media content208is stored in the metadata queue120.

After the metadata has been stored in the metadata queue120at210, the table of contents (TOC) object122generates a table of contents based on the metadata in the metadata queue120at212. The table of contents and the metadata are stored in the media file124at216.

As previously mentioned, media content decoded at206is analyzed at208, but the decoded media content is also encoded at214. The media content may be encoded in the format in which it was provided to the computer104, or may be encoded in another format. At216, the encoded media content is stored in the media file124along with the table of contents and the metadata. If the media content is to be stored in its original format at216, then one skilled in the art will recognize that encoding at214is not necessary as the media content stored in the media content queue106at204can be stored directly to the media file124at216.

The capturing and analyzing inFIG. 2may occur simultaneously. For example, if a 1 hour long video clip is on a tape based Digital Video (DV)-camcorder, it will take approximately 1 hour to provide all of the media content to the computer104at202. Initially, a first minute of the video clip can be provided to the computer104at202and stored in the media content queue106at204. The first minute in the queue106is then decoded at206by the transcoder108while the second minute of the video clip is being stored in the media content queue106at204. The uncompressed first minute of the video clip is then analyzed by the extensible analysis object112at208and encoded at214by the transcoder108. One operation or execution may take more time than another, but they may be performed at least partially simultaneously. Previewing the first minute of the video clip at218, analyzing the first minute and generating metadata at208, and storing metadata related to the first minute at210may all occur at the same time as214, encoding the media content. As the second minute of the video clip is captured and analyzed by the system, it has a similar relationship to the third minute of the video clip. It is also important to note that one series of operations or executions (i.e., the encoding of the media content versus the analyzing of the media content) of the operation may be faster or slower than another series. When both branches occur faster than the media content is provided to the computer at202, the user experience is not significantly impacted. If the analysis takes substantially longer than the encoding, then the user experience may be impacted (i.e., the user may be required to wait to manipulate the media file until the analysis has completed).

FIG. 3shows one example of a general purpose computing device in the form of a computer130. In one embodiment of the invention, a computer such as the computer130is suitable for use in the other figures illustrated and described herein. Computer130has one or more processors or processing units132and a system memory134. In the illustrated embodiment, a system bus136couples various system components including the system memory134to the processors132. The bus136represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example, and not limitation, such architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus also known as Mezzanine bus.

The computer130typically has at least some form of computer readable media. Computer readable media, which include both volatile and nonvolatile media, removable and non-removable media, may be any available medium that may be accessed by computer130. By way of example and not limitation, computer readable media comprise computer storage media and communication media. Computer storage media include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. For example, computer storage media include RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that may be used to store the desired information and that may be accessed by computer130. Communication media typically embody computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism and include any information delivery media. Those skilled in the art are familiar with the modulated data signal, which has one or more of its characteristics set or changed in such a manner as to encode information in the signal. Wired media, such as a wired network or direct-wired connection, and wireless media, such as acoustic, RF, infrared, and other wireless media, are examples of communication media. Combinations of any of the above are also included within the scope of computer readable media.

The system memory134includes computer storage media in the form of removable and/or non-removable, volatile and/or nonvolatile memory. In the illustrated embodiment, system memory134includes read only memory (ROM)138and random access memory (RAM)140. A basic input/output system142(BIOS), containing the basic routines that help to transfer information between elements within computer130, such as during start-up, is typically stored in ROM138. RAM140typically contains data and/or program modules that are immediately accessible to and/or presently being operated on by processing unit132. By way of example, and not limitation,FIG. 3illustrates operating system144, application programs146, other program modules148, and program data150.

The computer130may also include other removable/non-removable, volatile/nonvolatile computer storage media. For example,FIG. 3illustrates a hard disk drive154that reads from or writes to non-removable, nonvolatile magnetic media.FIG. 3also shows a magnetic disk drive156that reads from or writes to a removable, nonvolatile magnetic disk158, and an optical disk drive160that reads from or writes to a removable, nonvolatile optical disk162such as a CD-ROM or other optical media. Other removable/non-removable, volatile/nonvolatile computer storage media that may be used in the exemplary operating environment include, but are not limited to, magnetic tape cassettes, flash memory cards, digital versatile disks, digital video tape, solid state RAM, solid state ROM, and the like. The hard disk drive154, and magnetic disk drive156and optical disk drive160are typically connected to the system bus136by a non-volatile memory interface, such as interface166.

The drives or other mass storage devices and their associated computer storage media discussed above and illustrated inFIG. 3, provide storage of computer readable instructions, data structures, program modules and other data for the computer130. InFIG. 3, for example, hard disk drive154is illustrated as storing operating system170, application programs172, other program modules174, and program data176. Note that these components may either be the same as or different from operating system144, application programs146, other program modules148, and program data150. Operating system170, application programs172, other program modules174, and program data176are given different numbers here to illustrate that, at a minimum, they are different copies.

A user may enter commands and information into computer130through input devices or user interface selection devices such as a keyboard180and a pointing device182(e.g., a mouse, trackball, pen, or touch pad). Other input devices (not shown) may include a microphone, joystick, game pad, satellite dish, scanner, or the like. These and other input devices are connected to processing unit132through a user input interface184that is coupled to system bus136, but may be connected by other interface and bus structures, such as a parallel port, game port, or a Universal Serial Bus (USB). A monitor188or other type of display device is also connected to system bus136via an interface, such as a video interface190. In addition to the monitor188, computers often include other peripheral output devices (not shown) such as a printer and speakers, which may be connected through an output peripheral interface (not shown).

The computer130may operate in a networked environment using logical connections to one or more remote computers, such as a remote computer194. The remote computer194may be a personal computer, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to computer130. The logical connections depicted inFIG. 3include a local area network (LAN)196and a wide area network (WAN)198, but may also include other networks. LAN136and/or WAN138may be a wired network, a wireless network, a combination thereof, and so on. Such networking environments are commonplace in offices, enterprise-wide computer networks, intranets, and global computer networks (e.g., the Internet).

When used in a local area networking environment, computer130is connected to the LAN196through a network interface or adapter186. When used in a wide area networking environment, computer130typically includes a modem178or other means for establishing communications over the WAN198, such as the Internet. The modem178, which may be internal or external, is connected to system bus136via the user input interface184, or other appropriate mechanism. In a networked environment, program modules depicted relative to computer130, or portions thereof, may be stored in a remote memory storage device (not shown). By way of example, and not limitation,FIG. 3illustrates remote application programs192as residing on the memory device. The network connections shown are exemplary and other means of establishing a communications link between the computers may be used.

Embodiments of the invention may be described in the general context of computer-executable instructions, such as program modules, executed by one or more computers or other devices. Generally, program modules include, but are not limited to, routines, programs, objects, components, and data structures that perform particular tasks or implement particular abstract data types. Embodiments of the invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.

An interface in the context of a software architecture includes a software module, component, code portion, or other sequence of computer-executable instructions. The interface includes, for example, a first module accessing a second module to perform computing tasks on behalf of the first module. The first and second modules include, in one example, application programming interfaces (APIs) such as provided by operating systems, component object model (COM) interfaces (e.g., for peer-to-peer application communication), and extensible markup language metadata interchange format (XMI) interfaces (e.g., for communication between web services).

The interface may be a tightly coupled, synchronous implementation such as in Java 2 Platform Enterprise Edition (J2EE), COM, or distributed COM (DCOM) examples. Alternatively or in addition, the interface may be a loosely coupled, asynchronous implementation such as in a web service (e.g., using the simple object access protocol). In general, the interface includes any combination of the following characteristics: tightly coupled, loosely coupled, synchronous, and asynchronous. Further, the interface may conform to a standard protocol, a proprietary protocol, or any combination of standard and proprietary protocols.

The interfaces described herein may all be part of a single interface or may be implemented as separate interfaces or any combination therein. The interfaces may execute locally or remotely to provide functionality. Further, the interfaces may include additional or less functionality than illustrated or described herein. In operation, computer130executes computer-executable instructions such as those illustrated in the figures to implement embodiments of the invention.

Embodiments of the invention may be implemented with computer-executable instructions. The computer-executable instructions may be organized into one or more computer-executable components or modules. Embodiments of the invention may be implemented with any number and organization of such components or modules. For example, embodiments of the invention are not limited to the specific computer-executable instructions or the specific components or modules illustrated in the figures and described herein. Other embodiments of the invention may include different computer-executable instructions or components having more or less functionality than illustrated and described herein.

Having described embodiments of the invention in detail, it will be apparent that modifications and variations are possible without departing from the scope of embodiments of the invention as defined in the appended claims. As various changes could be made in the above constructions, products, and methods without departing from the scope of embodiments of the invention, it is intended that all matter contained in the above description and shown in the accompanying drawings shall be interpreted as illustrative and not in a limiting sense.