Video annotation using deep network architectures

A method includes receiving, by a processing device of a content sharing platform, a video content, selecting at least one video frame from the video content, subsampling the at least one video frame to generate a first representation of the at least one video frame, selecting a sub-region of the at least one video frame to generate a second representation of the at least one video frame, and applying a convolutional neuron network to the first and second representations of the at least one video frame to generate an annotation for the video content.

TECHNICAL FIELD

This disclosure relates to the field of video content sharing and, in particular, to annotate video contents based on video frames using a neuron network.

BACKGROUND

Large amounts of video content are made available on the Internet for users to browse and view. To search for a particular video clip, a user of the video content may enter one or more keywords as a query to a search engine to initiate a search process. The search engine may seek out video clips that match the request of the user based on certain criteria, and return to the user with a ranked list of video clips related to the query. One way to perform the search is to match the entered keywords with meta-data associated with each video clip. The meta-data may include the author, actors, and a short description of the video. However, not all video clips are associated with meta-data. Additionally, some video clips are associated with incomplete meta-data. For example, the meta-data of a video clip may just include the author, but may not provide textual description that may be helpful for the search. Thus, there is a significant amount of video content that needs to be annotated or associated with correct meta-data.

SUMMARY

Implementations of the disclosure may include a method that provides for receiving, by a processing device of a content sharing platform, a video content, selecting at least one video frame from the video content, subsampling the at least one video frame to generate a first representation of the at least one video frame, selecting a sub-region of the at least one video frame to generate a second representation of the at least one video frame, and applying a convolutional neuron network to the first and second representations of the at least one video frame to generate an annotation for the video content.

In one implementation, the method further provides that the first representation is a lower-resolution representation of the at least one video frame, and the second representation is a fovea representation that covers a smaller region than, but at a same spatial sampling rate as, the at least one video frame.

Implementations of the disclosure may include a machine-readable non-transitory storage medium storing instructions which, when executed, cause a processing device to perform operations including receiving a video content, selecting a plurality of video frames from the video content, applying a convolutional neuron network to the plurality of video frames to generate an annotation for the video content; and making the annotated video content available for search on a content sharing platform.

Implementation of the disclosure may include a system including a memory and a processor communicably coupled to the memory, the processor to execute content sharing platform. The processor receive a video content, select at least one video frame from the video content, subsample the at least one video frame to generate a first representation of the at least one video frame, select a sub-region of the at least one video frame to generate a second representation of the at least one video frame, and apply a convolutional neuron network to the first and second representations of the at least one video frame to generate an annotation for the video content.

DETAILED DESCRIPTION

A human observer may view a video clip and manually enter keyword annotations to create searchable meta-data for the video clip. Although manual annotation might be accurate, manual annotation of video is very expensive and is not fast enough to deal with the rapid growth of video content on the Internet. Therefore, there is a need for system and method that automatically derive meta-data containing descriptive annotations directly from video frames of video clips.

A video clip may include a stack of video frames (or images) each of which is to be displayed at a frame rate. For example, video frames may be played at 30 frames per second or at a higher rate. Each video frame may include a two-dimensional array of pixels, and each pixel may include pixel intensities. The pixel intensities may be grey levels for black and white video frames or color intensities (red, green, blue) for colored video frames. The number of pixels in a video frame denotes the resolution of the video frame, and the number of bits used for each pixel denotes the pixel depth or precision. Thus, a video clip may be represented by a volume of pixels in a three-dimensional space (x, y, t), in which x and y are the spatial coordinates, and t is the temporal axis. Implementations of the present disclosure include systems and methods that derive annotation keywords from the three-dimensional volume of pixels representing a video clip.

Implementations of the present disclosure include systems and methods that apply deep learning architectures directly on pixels of video frames to generate annotations. Compared to feature-based approaches where spatial and temporal features (e.g., edges) are first extracted from video frames, and then the extracted features are fed into a learning architecture, implementations of the present disclosure apply deep learning architecture directly to pixels of video frames without the intermediate feature extraction process, and achieve superior results than feature-based approaches.

Convolutional neuron network (CNN) is a type of deep learning architectures. A deep learning architecture refers to algorithms in machine learning that attempt to learn in multiple levels, corresponding to different levels of abstraction. CNN, as a type of deep learning architecture, is a feed-forward artificial neural network in which the individual neurons respond to overlapping regions in the visual field. CNN may include a filter layer, a pooling layer, and a connected neuron network.

The filter layer may include convolution operators that may convolve the input video frames VF(i, m, n, t) with a kernel K (i, m, n, t) or VF*K (i, m, n, t), where i represents pixel intensity index (imax=1 for grey scale, and imax=3 for colored), m and n are spatial indices, and t is the temporal index (or frame numbers). A kernel refers to the filter as applied to the video frames. In an implementation, the size of kernel K may be small compared to the size of the video frame to achieve compact computation. For example, for a video frame of 170×170×3×T, where T is a number representing total frames, the kernel may be 3×3×3×4. In an implementation, the filter layer may include a number of convolution operators that may convolve different kernels with the input to achieve different filtering results. The outcomes of the filter layer may be fed into a subsequent pooling layer or another filter layer.

The pooling layer may include average operators and max-pooling operators to down sample the input video frames to smaller sizes. The average operators may produce an average value within a particular range. In an implementation, the average operator may be a simple average of the pixel intensities within the particular range. In other implementations, the average operator may specify weights for each pixel and produce a weighted-average of the pixel intensities. Max-pooling operators may produce an outcome of the max intensity within a particular range. Like kernels used in filter layers, operators used in pooling layers may also be small compared to the size of video frames to achieve compact computation. For example, the average and/or pooling-max operator may be 3×2×2×3. The size of the pooling layer output may be smaller than that of its input, each data point of the output representing an averaged value of a sub-region of the input.

The input of a CNN may be subject to multiple filter layers and/or pooling layers prior to applying the neuron network layer to the filtered and pooled input. Each pooling layer may reduce the size of input to the pooling layer. Thus, after several layers of filtering and pooling, the input to the neuron network may be reduced to a small amount of data. The neuron network may be a fully-connected neuron network, including hidden layers of neurons that can compute values from inputs by feeding information through the network and a layer of softmax activation functions that calculate the layer's output from its net input in a manner similar to biologically plausible approximation to the maximum operation. Parameters of the fully-connected neuron network may be trained on labeled training data. In an implementation, the training data may be labeled manually. For example, human observers may view a set of video clips and manually create training meta-data including annotations to each video clips.

In an alternative embodiment, the training data may be labeled automatically to generate a large amount of training data. Unlike those feature-based neuron network approaches where spatial and temporal features (such as edges in the intensity images) in the training video frames are also labeled manually, the deep learning architectures as used in implementations are applied to the pixels directly. Because pixels do not need labeling, large amount of training data can be generated automatically from the large amount of video contents that are already in existence. For example, for action video such as “mountain biking,” there are thousands of video clips that are annotated with the keyword “mountain biking” on different websites. These video clips may be converted into training data through processing steps such as cropping to a predetermined size and pixel intensity normalization. In this way, the training data may be easily generated for different classes of video clips and at very fine granularity of keywords.

FIG. 1illustrates a system architecture100in which aspects of the disclosure can be implemented. The architecture100includes a plurality of data stores102A-102Z, a network106, a content sharing platform108, and client devices110A through110Z. In an implementation, data stores102A-102Z may be repositories of video contents that are accessible through network106. Data stores may include multiple storage components (e.g., multiple drives or multiple databases) that may also span multiple computing devices (e.g., multiple server computers).

Content sharing platform may be connected to data stores102A-102Z and/or client devices110A-110Z via a network106. Network106may include a public network (e.g., the Internet), a private network (e.g., a local area network (LAN) or wide area network (WAN)), a wired network (e.g., Ethernet network), a wireless network (e.g., an 802.11 network or a Wi-Fi network), a cellular network (e.g., a Long Term Evolution (LTE) network), routers, hubs, switches, server computers, and/or a combination thereof. In one implementation, the data store106may be a memory (e.g., random access memory), a cache, a drive (e.g., a hard drive), a flash drive, a database system, or another type of component or device capable of storing data.

The client devices110A through110Z may each include computing devices such as personal computers (PCs), laptops, mobile phones, smart phones, tablet computers, netbook computers, etc. In some implementations, client device110A through110Z may also be referred to as “user devices.” Each client device may include a media viewer (not shown). In one implementation, the media viewers may be applications that allow users to view content, such as images, videos, web pages, documents, etc. For example, the media viewer may be a web browser that can access, retrieve, present, and/or navigate content (e.g., web pages such as Hyper Text Markup Language (HTML) pages, digital media items, etc.) served by a web server. The media viewer may render, display, and/or present the content (e.g., a web page, a media viewer) to a user. The media viewer may also display an embedded media player (e.g., a Flash® player or an HTML5 player) that is embedded in a web page (e.g., a web page that may provide information about a product sold by an online merchant).

In another example, the media viewer may be a standalone application (e.g., a mobile app) that allows users to view digital media items (e.g., digital videos, digital images, electronic books, etc.).

The media viewers may be provided to the client devices110A-110Z by content sharing platform108. For example, the media viewers may be applications that are downloaded from the content sharing platform108or a third-party app store.

In general, functions described in one implementation as being performed by the content sharing platform108can also be performed on the client devices110A-110Z in other implementations if appropriate. In addition, the functionality attributed to a particular component can be performed by different or multiple components operating together. The content sharing platform108can also be accessed as a service provided to other systems or devices through appropriate application programming interfaces, and thus is not limited to use in websites.

In one implementation, the content sharing platform108may include one or more computing devices (such as a rackmount server, a router computer, a server computer, a personal computer, a mainframe computer, a laptop computer, a tablet computer, a desktop computer, etc.), data stores (e.g., hard disks, memories, databases), networks, software components, and/or hardware components that may be used to provide a user with access to media items and/or provide the media items to the user. For example, the content sharing platform108may allow a user to consume, upload, search for, approve of (“like”), dislike, and/or comment on media items. The content sharing platform108may also include a website (e.g., a webpage) that may be used to provide a user with access to the media items such as video contents.

A media item may be consumed via the Internet and/or via a mobile device application. For brevity and simplicity, an online video (also hereinafter referred to as a video) is used as an example of a media item throughout this document. As used herein, “media,” media item,” “online media item,” “digital media,” “digital media item,” “content,” and “content item” can include an electronic file that can be executed or loaded using software, firmware or hardware configured to present the digital media item to an entity. In one implementation, the content sharing platform108may store hyperlinks to the media items stored on the data stores102A-102Z.

In one implementation, the content sharing platform108includes an annotation subsystem112that converts raw video content (e.g., content stored in data stores102and/or received from client devices110) into annotated video content to facilitate video classification, video searching, ad targeting, spam and abuse detection, content rating, etc.

FIG. 2illustrates operation of an annotation subsystem112according to an implementation of the disclosure. Annotation subsystem112may be a computer system700discussed in more detail below in conjunction withFIG. 7.

As shown inFIG. 2, raw video contents202may include hyperlinks204A-204Z stored on the content sharing platform108. Hyperlinks204A-204Z may provide links to raw videos stored in data stores102over the network. These raw videos may not be associated with annotations in the form of meta-data, or alternatively, the annotations associated these raw videos may need to be updated. Annotation subsystem112associates raw video contents (such as those without annotation keywords or tags) with annotations or updated annotations. In an implementation, the annotation subsystem112may retrieve whole, or part of, a video clip for processing. The annotation subsystem112then may analyze the video frames of the video clip and generate annotations for the video clip. The annotation subsystem112may generate annotations for each of the video clips on the content sharing platform108so that each of the hyperlinks204A-204Z may be associated with one or more annotations210A-210Z. In one implementation, the one or more annotations210A-210Z may include keywords indicating the nature of the videos linked to hyperlinks204A-204Z. In another implementation, the one or more annotations210A-210Z may include tags indicating that videos linked by hyperlinks204A-204Z belong to pre-specified categories. The annotated video contents206may be ready to be searchable using annotations and consumed by users of the content sharing platform108through client devices and network.

Implementations of the present disclosure may include methods that use deep learning architectures for generating annotations for raw video contents. In an implementation, the deep learning architectures may include a convolutional neuron network (CNN) that may be trained directly on pixel intensities contained in video frames of video contents. Compared to feature-based neuron network approaches, CNN systems that directly analyze pixels may have the advantage of ease to generate a large amount of training data and more robust results. CNN systems implemented according to aspects of this disclosure may also have the advantage of flexibility to scale in spatial and temporal dimensions. Since the CNN systems work directly on pixels, they may use kernels and windows of weights of different sizes as the resolution and precision permit. Because of this flexibility of scaling, implementations may be customized to specific needs and achieve the desired results faster.

FIGS. 3A-3Dillustrate different implementations of applying CNN systems to video contents (e.g., video clips) according to implementations of the disclosure.FIG. 3Aillustrates the application of a convolutional neuron network310to a video clip300according to an implementation. The video content300may be a raw video clip including a stack of video frames (illustrated inFIGS. 3A-3Das F1-F16) temporally arranged along the t axis at a frame rate (such as 30 frames per second). Since the frame rate is fixed, the index to video frames indicates temporal dimension of the video frames. Each video frame may include a 2D array of pixels where each pixel may include one or more pixel intensity channels. For example, pixels of colored video contents may include three pixel intensities denoting red, green, and blue. Alternative color formats such as YUV may be derived from the RGB format. Pixels, of black and white video contents, may include one channel representing grey levels.

CNN310that processes video content300may include a first component of filter and/or pooling layers312and a second component of a connected neuron network314. First component312may include a plurality of filter layers and pooling layers302.1-302.6. Filter layers may convolve kernels with input data, and pooling layers may average and down sample the input data. Connected neuron network314may include hidden layers304,306, and a softmax layer308. Parameters of hidden layers304,306and softmax layer308may be trained on labeled training data so that the connected neuron network314may generate annotations from the outputs of filter/pooling layers312.

In an implementation, as shown inFIG. 3A, the input data to the CNN system may be a single video frame of the video content. Therefore, the input data as a single video frame may include a 2D array of pixels which may undergo filter and/or pooling layers302.1-302.6. The output of the filter and/or pooling layers302.1-302.6may be supplied to connected neuron network314to generate annotations associated with video content300. In an implementation, the generated annotations may include a list of keywords ranked according to a likelihood value of a keyword being associated with the video content300. For better results, parameters of hidden layers304,306, and softmax layer308are trained from singular video frames that were labeled with annotations.

CNN systems trained based on singular video frames, as shown inFIG. 3A, take into consideration only spatial information within a video frame but do not utilize the temporal correlation information between frames. When object movements are captured in a video clip, information about object movements may be captured in the form of temporal pixel correlation between nearby video frames. This correlational information may be encoded through a number of frames.

FIG. 3Billustrates an early fusion model that takes advantage of multiple video frames according to an implementation of the disclosure. Instead of a single video frame, a stack of temporally correlated video frames may be used as the input data to CNN systems. In an implementation, as shown inFIG. 3B, a stack of consecutively indexed video frames (e.g., frames F7-F11) may be used as input. Alternatively, temporally-correlated video frames may be stacked together as input. For example, every other frame such as frames F7, F9, F11may be stacked together as input data to the CNN system310. When video frames are stacked together, the input data may be viewed as a 3D data volume including spatial dimensions x and y, and temporal dimension t. Likewise, kernels in filter layers may also be 3D. In an implementation, the kernel may have dimensions of 3×3×3 for each pixel channel. Similarly, the weighted average operators for pooling layers may also be 3D. Outputs from filter and/or pooling layers312may be fed into a connected neuron network314to generate annotations for the video content300. In an implementation, the generated annotations may include a list of keywords ranked according to a likelihood value of a keyword being associated with the video content300. Parameters of the connected neuron network314may have been trained based on stacked video frames that had been previously labeled.

Rather than the early fusion of temporal correlation information, implementations of the disclosure may fuse the temporal information at late stages. FIG.3C illustrates a late fusion model that takes advantage of multiple video frames according to an implementation of the disclosure. As shown inFIG. 3C, two or more non-consecutive video frames (e.g., frames F1, F7, F16) may be used as input. In addition, instead of processing a stack of video frames together, filter and/or pooling layers of the CNN system as shown inFIG. 3Cprocess a number of video frames each individually, i.e., without taking into consideration their temporal correlation at the initial stage. Each of the video frames (e.g., F1, F7, F16) may undergo the same filter and/or pooling layers302.1-302.6. The outcomes of each video frame F1, F7, F16, may be fed into the connected neuron network314for a late fusion that may then take into consideration temporal correlation among video frames F1, F7, F16.

In one implementation, the early and late fusions may be combined together for form a hybrid fusion model.FIG. 3Dillustrates a hybrid fusion model that combines early and late fusions according to an implementation of the disclosure. Similar to the early fusion model, video frames of video content300may be grouped into different groups. In one implementation, as shown inFIG. 3D, frames F1-F4may be grouped into a first group, frames F6-F9may be grouped into a second group, and frames F13-F16may be grouped into a third group. Each group may be similarly processed by filter and pooling layers302.1-302.6in a manner similar to the early fusion model as shown inFIG. 3B. The outcomes from processing the three groups of video frames may be fed into the connected neuron network314to generate annotations for the video content300. Parameters of the connected neuron network314may have been trained based on a combination of grouped video frames that had been previously labeled.

In one implementation, the fusion of temporal correlation among video frames may be developed progressively over layers of filter and pooling so that higher layers may progressively gain access to more global information in both spatial and temporal dimensions.FIG. 3Eillustrates a progressive fusion model according to an implementation of the present disclosure. As shown inFIG. 3E, filter and/or pooling layers may be applied to video frames contained in video content300. Lower layers or the first level layers may include groups of layers302.1,302.2that may have a larger temporal extent. For example, each group of layers302.1,302.2, as shown in the example implementation ofFIG. 3E, may have a temporal extent T=4 and a stride ST=2. Thus, each group of the first level of filter and/or pooling layers302.1,302.2may cover four video frames. A middle layer or the second level layers may include layers302.3,302.4that may have a smaller temporal extent. For example, layers302.3,302.4, as shown in the example implementation ofFIG. 3E, may have a temporal extent T=2 and a stride ST=1. Thus, each group of the second level of filter and/or pooling layers302.3,302.4may cover six video frames. The higher layer of the third level of filter may include layers302.5,302.6that may have a temporal extent T=2 and a stride ST=0, and cover all eight video frames. In this way, the temporal correlation information is progressively incorporated through filter and/or pooling layers302.1-302.6. The outcome of the progressive fusion of video frames may be fed into the connected neuron network314to generate annotations for video content300.

As discussed in conjunction withFIG. 3E, the convolution layer and the pooling layer at a lower level may be applied to a first number of video frames while the convolution layer and a second pooling layer at a higher level may be applied to a second number of video frames. For example, at the lower level, the convolution layer and the pooling layer may be applied to four video frames while at the higher level, they may cover six video frames.

Implementations discussed so far have viewed the input to CNN systems as a whole frame. The whole video frame may include a large array of pixels such as an array of 170×170 pixels that require a lot of computational time to train the CNN. To reduce the computational burden while still achieving good annotation results, the video frames may be decomposed into different components. Each of the decomposed components may then be analyzed by the CNN.FIG. 4illustrates an example decomposition of the input video frames according to an implementation of the disclosure. In an implementation, the first decomposition component is a fovea representation that is a central region cropped from each video frame contained in video content300. The size of the fovea representation may be much smaller than that of the whole video frame. Since videos are typically shot such that actions are taken in the middle of video frames to grab viewer's attention, the small fovea regions still include rich motion information. The second decomposition component may include a spatial sub-sample of the video frames. In an implementation, every other pixel may be discarded so that the sub-sample component contains only one fourth of pixels contained in a whole video frame. In this way, the total number of pixels that need to be processed may be significantly reduced while information about the main characteristics of video frames is preserved.

As shown inFIG. 4, first and second decomposition components may be fed into a CNN system that may generate annotations for the video contents. The decomposed video frames may be processed in the manners similarly to those implementations described inFIGS. 3A-3E. Thus, each decomposition component may be treated as a single frame, early fusion, late fusion, hybrid fusion, and progressive fusion models. Parameters of the CNN may be trained according to these models as well.

As discussed above, CNN systems can be used different implementations to annotate video content.FIGS. 5 and 6illustrate flow diagrams of methods for annotating video content, according to some implementations of the disclosure. The method may be performed by processing logic that comprises hardware (e.g., circuitry, dedicated logic, programmable logic, microcode, etc.), software (e.g., instructions run on a processing device to perform hardware simulation), or a combination thereof.

Referring toFIG. 5, at502, the annotation subsystem112may retrieve a video content from a data store. The video content may include a plurality of video frames that are not associated with annotations. At504, the annotation subsystem112may select a stack (or subset) of video frames from the video content. Depending of the model used, the stack may include one or more video frames. A single frame model may need one video frame, while other models as described above may need more than one video frame. At506, the annotation subsystem112may apply a previously-trained convolutional neuron network to the stack of video frames to generate annotations for the video content. The CNN may be trained manually or automatically as discussed previously. The annotations may include a list of keywords associated with the video content. At508, the annotation subsystem112may make the annotated video content available on a content sharing platform by associating keywords with a hyperlink of the video content.

FIG. 6illustrates a flow diagram of a method to perform annotating video content according to another implementation of the disclosure. At602, the annotation subsystem112may retrieve a video content from a data store. The video content may include a plurality of video frames that are not associated with annotations. At604, the annotation subsystem112may select a stack (or subset) of video frames from the video content. Depending of the model used, the stack may include one or more video frames. The stack of video frames may then be decomposed into two components as discussed in the following606,608. At606, the annotation subsystem112may spatially subsample video frames to generate a first representation of the video frames. In one implementation, the subsampling may be uniform across the spatial extent of the video frames. In another implementation, the subsampling may be random across the spatial extent of the video frames. The outcome of the sub sampling may be smaller than the original video frame. In other words, the number of pixels in the outcome is a small proportion of the number of original pixels. In one implementation, the number of pixels in the outcome is less than 25 percent of the original pixels.

At608, the annotation subsystem112may also select a sub-region from video frames to generate a second representation of the video frames. The sub-region may be a small portion of the whole video frame. In an implementation, the sub-region may be one fourth of the original video frame while the sub-region has the same spatial sample rate as the video frame. Thus, the sub-region may include one fourth of pixels as the whole video frame. In an implementation, the select sub-region may be in the center of the video frame and is referred to as the fovea region. When multiple video frames are stacked up, the fovea region may include primary actions of the video. In an alternative implementation, the sub-region may be located at a non-central location. At610, the annotation subsystem112may apply a convolutional neuron network to the first and second representations of the video frames, respectively, to generate annotations for the video content. Different models as described inFIGS. 3A-3Emay be utilized in CNN systems. In one implementation, the first and second representations are derived from a single video frame. In other implementations, the first and second representations are derived from a plurality of video frames so that the early fusion, late fusion, hybrid fusion, and progressive fusion models as described inFIGS. 3B-3Emay be used. The annotated videos generated from implementations of the disclosure may be made available on a content search platform for consumers to consume.

The computer system700may further include a network interface device722. The computer system700also may include a video display unit710(e.g., a liquid crystal display (LCD), a cathode ray tube (CRT), or a touch screen), an alphanumeric input device712(e.g., a keyboard), a cursor control device714(e.g., a mouse), and a signal generation device720(e.g., a speaker).

The data storage device718may include a computer-readable storage medium724on which is stored one or more sets of instructions726(e.g., software) embodying any one or more of the methodologies or functions described herein (e.g., instructions of the annotation subsystem112). The instructions726may also reside, completely or at least partially, within the main memory704and/or within the processor702during execution thereof by the computer system700, the main memory704and the processor702also constituting computer-readable storage media. The instructions726may further be transmitted or received over a network774via the network interface device722.