Patent Description:
Animation is an extremely large business worldwide and a major offering of many of the largest media companies. However, animated videos typically contain very limited meta-data and, therefore, efficient search and retrieval of specific content is not always possible. For example, a key component in animated media is the animated characters themselves. Indeed, characters in the animated videos must first be indexed, e.g., detected, classified, and annotated, in order to enable efficient search and retrieval of those characters within the animated video.

Various services can leverage artificial intelligence or machine learning for image understanding. However, these services typically rely on extensive manual labeling. For example, character recognition in an animated video currently involves manually drawing bounding boxes around each character and tagging (or labeling) the character contained therein, e.g., with the name of the character. This manual annotation process is repeated for each character of every frame of a multi-frame animated video. Unfortunately, this manual annotation process is tedious and severely limits scalability of these services.

Overall, the examples herein of some prior or related systems and their associated limitations are intended to be illustrative and not exclusive. Upon reading the following, other limitations of existing or prior systems will become apparent to those of skill in the art. <CIT> describes a kind of improved online Boosting method for tracking target, and for the low problem of the Haar feature efficiencies that Boosting algorithms use. <NPL> discusses object detection.

Hanss314: "BFB <NUM> except with an object detection neural network", https://www. com/ watch?v=x0GrFthY9_M shows object detection applied on animated videos.

One or more embodiments described herein, among other benefits, solve one or more of the foregoing or other problems in the art by providing systems, methods, and non-transitory computer readable media that can automatically detect and group instances (or occurrences) of characters in a multi-frame animated media file such that each group contains images associated with a single character. The character groups themselves can then be labeled and used to train an image classification model for automatically classifying the animated characters in subsequent multi-frame animated media files.

As will be realized, the invention is capable of modifications in various aspects, all without departing from the scope of the present invention.

This Overview is provided to introduce a selection of concepts in a simplified form that are further described below in the Technical Disclosure. It may be understood that this Overview is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Additional features and advantages of the present application will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by the practice of such example embodiments.

In order to describe the manner in which the above-recited and other advantages and features can be obtained, a more particular description is set forth and will be rendered by reference to specific examples thereof which are illustrated in the appended drawings. Understanding that these drawings depict only typical examples and are not therefore to be considered to be limiting of its scope, implementations will be described and explained with additional specificity and detail through the use of the accompanying drawings.

The drawings have not necessarily been drawn to scale. Similarly, some components and/or operations may be separated into different blocks or combined into a single block for the purposes of discussion of some of the embodiments of the present technology. Moreover, while the technology is amenable to various modifications and alternative forms, specific embodiments have been shown by way of example in the drawings and are described in detail below. The intention, however, is not to limit the technology to the particular embodiments described. On the contrary, the technology is intended to cover all modifications, equivalents, and alternatives falling within the scope of the technology as defined by the appended claims.

Examples are discussed in detail below. While specific implementations are discussed, it should be understood that this is done for illustration purposes only. A person skilled in the relevant art will recognize that other components and configurations may be used The implementations may include machine-implemented methods, computing devices, or computer readable medium.

Recognizing animated characters in video can be challenging for a number of reasons such as the unorthodox nature of animated characters themselves. Indeed, animated characters can come in many different forms, shapes, sizes, etc. In many cases, content producers, e.g., companies generating or manipulating animated media content, would like to index the characters included in their animated media content. However, as noted above, this is presently a very difficult and non-scalable process that requires manually annotating each character in every frame of a multi-frame animated media file.

The technology described herein is directed to a media indexer including a character recognition engine that can automatically detect and group instances (or occurrences) of characters in a multi-frame animated media file such that each group contains images associated with a single character. The character groups themselves are then labeled and the labeled groups are used to train an image classification model for automatically classifying the animated characters in subsequent multi-frame animated media files.

Various technical effects can be enabled by the techniques discussed herein. Among other benefits, the techniques discussed herein provide a scalable solution for training image classification models with minimal comprise to character detection or character classification accuracy. Additionally, the use of keyframes reduces the amount of data that needs to be processed while keeping the variance of data high. Furthermore, automated character recognition eliminates the need to manually annotate bounding boxes, and automated grouping of the characters yields accurate annotations with substantially reduced manual effort, e.g., semi-supervised training via group labeling as opposed to character-by-character annotation.

As used herein, the term "animated character" refers to an object that exhibits humanlike traits contained or detected in an animated multi-frame animated media file. For example, an "animated character" can be an animate or inanimate anthropomorphic object that exhibits any human form or attribute including, but not limited to, a human trait, emotion, intention, etc..

A general overview and example architecture of an animated character recognition and indexing framework is described for training an AI-based image classification model in relation to <FIG>. <FIG> then depicts an example whereby the animated character recognition and indexing framework applies (and re-trains or refines as necessary) the trained AI-based image classification model. Thereafter, a more detailed description of the components and processes of the animated character recognition and indexing framework are provided in relation to the subsequent figures.

<FIG> depicts a block diagram illustrating an example animated character recognition and indexing framework <NUM> for training an AI-based image classification model to automatically classify characters in a multi-frame animated media file for indexing, according to some implementations. Indeed, the example animated character recognition and indexing framework <NUM> includes a media indexer service <NUM> that can automatically detect and group instances (or occurrences) of characters in the media file such that each group contains images associated with a single character. The character groups are then identified (or recognized) and labeled accordingly. As shown in the example of <FIG>, the labeled character groups (or grouped character training data) can then be utilized to train the AI-based image classification model to automatically classify the animated characters in subsequent multi-frame animated media files.

As illustrated in the example of <FIG>, the animated character recognition and indexing framework <NUM> includes an image classifier <NUM>, a media indexer <NUM> and a user <NUM> operating a computing system <NUM> that can provide user input to manually label (or recognize) the character groups. Additional or fewer systems or components are possible.

The image classifier <NUM> can be any image classifier of image classification service. In some implementations, the image classifier <NUM> can be embodied by an Azure Custom Vision Service provided by Microsoft. The Custom Vision Service uses a machine learning algorithm to apply labels to images. A developer typically submits groups of labeled images that feature and lack the characteristics in question. The machine learning algorithm uses the submitted data for training and calculates its own accuracy by testing itself on those same images. Once the machine learning algorithm (or model) is trained, the image classifier <NUM> can test, retrain, and use the model to classify new images.

As illustrated in the example of <FIG> and <FIG>, the media indexer <NUM> includes a character recognition engine <NUM>, a media indexer database <NUM>, and an indexing engine <NUM>.

The character recognition engine <NUM> includes a keyframe selection module <NUM>, a character detection module <NUM>, a character grouping module <NUM>, and a group labeling module <NUM>. The functions represented by the components, modules, managers and/or engines of character recognition engine <NUM> can be implemented individually or in any combination thereof, partially or wholly, in hardware, software, or a combination of hardware and software. Additionally, although illustrated as discrete components, the operation and functionality of the components, modules, managers and/or engines of the character recognition engine <NUM> can be, partially or wholly, integrated within other components of the animated character recognition and indexing framework <NUM>.

In operation, a non-indexed (or unstructured) multi-frame animated media file 105a is fed to the media indexer <NUM> for character recognition and indexing. The media indexer <NUM> includes a character recognition engine <NUM>, a media indexer database <NUM>, and an indexing engine <NUM>. Additional or fewer systems or components are possible.

The keyframe selection module <NUM> is configured to select or otherwise identify a small subset of the total frames of a multi-frame animated media file to reduce computational complexity of the character recognition process with minimal or limited effect on accuracy. Indeed, the keyframe selection module <NUM> is configured to identify and select important or significant frames (e.g., frames with the highest likelihood of observing characters) from the multi-frame animated media file. In some implementations, the keyframes are determined based at least in part, on their individual significance in determining a micro-scene or fraction of a shot. In some implementations, each frame can be assigned a significance value and frames having a significance value greater than a threshold value are selected as keyframes. Alternatively, or additionally, a percentage of the total frames, e.g., top one percent, of the frames with the highest rated significance value can be selected as keyframes.

As discussed herein, the keyframes typically constitute a small fraction, e.g., one percent of the total frames in the multi-frame animated media file, e.g., animated video. However, the performance difference between labeling each of the frames in the multi-frame animated media file versus labeling just the keyframes is nominal for the purposes of detecting each of the characters in the multi-frame animated media file. Thus, the keyframes allow the media indexer <NUM> to maintain character detection accuracy while simultaneously reducing computation complexity.

The character detection module <NUM> is configured to process or analyze the keyframes to detect (or propose) instances (or occurrences) of characters in the keyframes of the multi-frame animated media file. Indeed, the character detection module <NUM> can process the keyframes and provide character region proposals (also referred to as bounding boxes). For example, the character detection module <NUM> can capture each character region proposal as an image.

As discussed herein, detection of animated characters can be difficult as the characters can take the form of almost any animate (e.g., person, animal, etc.) or inanimate (e.g., robot, car, candle, etc.) object. Accordingly, in some implementations, the character detection module <NUM> includes an object detection model trained to detect bounding boxes of animated characters in different styles, themes, etc., (e.g., car, persons, robots, etc.).

In some implementations, the character detection module <NUM> can be trained to detect objects that exhibit humanlike traits. That is, the character detection module <NUM> is designed to detect any anthropomorphic object within the keyframes. As discussed herein, the term "anthropomorphic object" refers to any animate or inanimate object that exhibits any human form or attributes including, but not limited to, human traits, emotions, intentions, etc..

The character grouping module <NUM> is configured to compare and group the character region proposals based on similarity of the images such that each group contains images associated with a single character. In some instances, more than one of the resulting character groups can be associated with the same character, e.g., a first group including images of Sponge Bob with a hat and a second group including images of Sponge Bob without a hat.

In some implementations, the character grouping module <NUM> applies a clustering algorithm using embeddings of the detected character region proposals to determine the character groups. Indeed, the character groups can be determined by embedding the features of the character region proposals (or images) into a feature space to simplify the image comparisons. An example illustrating a method of applying a clustering algorithm including embedding the character region proposals (or images) into the feature space and comparing the embeddings to identify the character groups is shown and discussed in greater detail with reference to <FIG>.

The group labeling module <NUM> is configured to label (annotate or classify) the character groups without the use of a classification model. As discussed herein, labeling the character groups is useful for initial training of a classification models as well as for refining trained classification models (as shown and discussed in greater detail with reference to <FIG>).

In some implementations, the group labeling module <NUM> can present each character group to the user <NUM> as a cluster of images. The character groups can then be classified with input from the user <NUM>. For example, the user <NUM> can provide an annotation or tag for the group. Alternatively, or additionally, the user <NUM> can provide canonical images of the characters that are expected to appear in the multi-frame animated media file. In such instances, the canonical characters can be compared to the character groups to identify and label the character groups. In other implementations, the user <NUM> can provide a movie or series name of the multi-frame animated media file. In such instances, the group labeling module <NUM> can query a data store, e.g., Satori (Microsoft knowledge graph), for information about the movie and/or series and to extract names of the characters and any available canonical images.

<FIG> depicts a block diagram illustrating the example animated character recognition and indexing framework <NUM> applying (and re-training as necessary) the AI-based image classification model trained in the example of <FIG>, according to some implementations. Indeed, the trained AI-based image classification model is trained to automatically recognize and index animated characters in multi-frame animated media file 106a. The multi-frame animated media file 106a is related (e.g., of a same series or with one or more overlapping characters) to the multi-frame animated media file 105a.

As discussed herein, in some implementations, a user can specify a trained AI-based image classification model to use for indexing a multi-frame animated media file. An example illustrating a graphical user interface including various menus for selecting the trained AI-based image classification model is shown and discussed in greater detail with reference to <FIG>.

In operation, the media indexer <NUM> can utilize the trained AI-based image classification model to classify character groups and refine (or tune) the trained AI-based image classification model using new grouped character training data, e.g., new characters or existing characters with new or different looks or features. As discussed herein, the media indexer <NUM> interfaces with the image classifier <NUM> to utilize, train, and/or refine the AI-based image classification model(s) <NUM>.

As discussed above, the image classifier <NUM> can be embodied by the Azure Custom Vision Service which can be applied per cluster (or character group). In some implementations, a smoothing operation can be applied to handle cases where a single character is split into two or more different clusters (or character groups), e.g., group including images of Sponge Bob with a hat and group including images of Sponge Bob without a hat. The smoothing operation is operable to consolidate the two or more different clusters (or character groups) and provide grouped character training data to refine the trained AI-based image classification model such that future classifications are classified as the same character.

<FIG> depicts a data flow diagram that graphically illustrates operations and the flow of data between modules of a media indexer <NUM>, according to some implementations. As shown in the example of <FIG>, the media indexer <NUM> includes the keyframe selection module <NUM>, the character detection module <NUM>, the character grouping module <NUM>, and the group labeling module <NUM> of <FIG> and <FIG>. Additional or fewer modules, components or engines are possible.

<FIG> depicts a flow diagram illustrating an example process <NUM> for indexing a multi-frame animated media file, e.g., animated video, using the automated character detection and grouping technique discussed herein, according to some implementations. The example process <NUM> may be performed in various implementations by a media indexer such as, for example, media indexer <NUM> of <FIG> and <FIG>, or one or more processors, modules, engines, or components associated therewith.

To begin, at <NUM>, the media indexer presents a user interface (UI) or application program interface (API). As discussed herein, the user can specify both a multi-frame animated media file to be indexed and an AI-based image classification model with which to index (if trained) or with which to train (if untrained). An example illustrating a graphical user interface including various menus for selecting the trained AI-based image classification model is shown and discussed in greater detail with reference to <FIG>.

At <NUM>, the media indexer receives a multi-frame animated media file, e.g., animated video, for indexing. At <NUM>, the media indexer extracts or identifies keyframes. At <NUM>, the media indexer detects characters in the keyframes. At <NUM>, the media indexer groups the characters that are automatically detected in a multi-frame animated media file. An example illustrating character grouping is shown and discussed in greater detail with reference to <FIG>. At <NUM>, the media indexer determines if a trained classification model is specified. If so, at <NUM>, the media indexer classifies the character groups using the trained classification model and, at <NUM>, smooths (or consolidates) the classified character groups.

Lastly, at <NUM>, the multi-frame animated media file, e.g., animated video, is indexed with recognized (classified) and unrecognized (unknow) characters. An example graphical user interface illustrating an indexed multi-frame animated media file with both recognized and unrecognize characters is shown in the example of <FIG>. As discussed herein, the user can specify or label the unrecognize character groups to refine the AI-based image classification model.

<FIG> depicts a flow diagram illustrating an example process <NUM> for training or refining an AI-based image classification model using grouped character training data, according to some implementations. The example process <NUM> may be performed in various implementations by a media indexer such as, for example, media indexer <NUM> of <FIG> and <FIG>, or one or more processors, modules, engines, or components associated therewith.

To begin, at <NUM>, the media indexer identifies (or otherwise obtains) label or classification information.

<FIG> depicts a flow diagram illustrating an example process <NUM> for grouping (or clustering) characters that are automatically detected in a multi-frame animated media file, according to some implementations. The example process <NUM> may be performed in various implementations by a media indexer such as, for example, media indexer <NUM> of <FIG> and <FIG>, or one or more processors, modules, engines, or components associated therewith.

To begin, at <NUM>, the media indexer identifies (or otherwise obtains) label or classification information for unknown (or unclassified) animated character groups. As discussed herein, the media indexer can identify label information, e.g., the name of the single animated character associated with each animated character group and classify (or annotate) the animated character groups with the identified label information resulting in at least one annotated animated character group.

At <NUM>, the media indexer collects the identified (or annotated) animated character groups in a media indexer database. Lastly, at <NUM>, the media indexer trains or refines an image classification model by feeding the annotated animated character groups to an image classifier to train an image classification model.

To begin, at <NUM>, the media indexer accesses a next identified character. As discussed herein, each character region proposal comprises a bounding box or subset of a keyframe containing a proposed animated character. At <NUM>, the media indexer extract features of the next identified character contained in the character region proposal and, at <NUM>, embeds the features in a feature space.

At decision <NUM>, the media indexer determines if more character region proposals have been identified and, if so, returns to step <NUM>. As discussed herein, multiple keyframes from a multi-frame animated media file are first identified. Each of keyframes can include one or more character region proposals. Once each character region proposal is travers, at <NUM>, the media indexer selects the groups of clusters character in the feature space. For example, the media indexer can determine a similarity between the character region proposals by comparing the embedded features within the feature space and apply a clustering algorithm to identify the animated character groups based on the determined similarity.

<FIG> depict various graphical user interfaces that can be presented to a user. Referring first to the example of <FIG> depicts a graphical user interface including various menus for selecting various options for uploading a video file, according to some implementations. More specifically, <FIG> depicts a graphical user interface including various menus for selecting various options for uploading a video file and (optionally) selecting a trained AI-based image classification model with which to index the video file (or alternatively to train).

Referring next to the example of <FIG> which depicts a graphical user interface illustrating an example video that has been indexed using the media indexer discussed herein. Indeed, the example of <FIG> illustrates instances of various different characters that have been identified, classified and indexed in an example video.

Similarly, <FIG> depicts a graphical user interface illustrating an example video that has been indexed using the media indexer discussed herein. More specifically,.

The example process <NUM> is similar to example process <NUM> except that example process <NUM> includes steps for style adaptation. For example, an AI-based image classification model can be trained using a first type (or style) of animation, e.g., computer generated graphics (CGI) and subsequently applied to an input including as second type (or style) of animation, e.g., hand drawn animations, without retraining model. Among other potential options, the keyframes can be adjusted or transformed (as shown in the example of <FIG>) or the extracted features can be transformed prior to embedding into the feature space (as shown in the example of <FIG>).

Referring again to <FIG>, in some implementations, an additional network for style adaptation can be added to the detector, e.g., character detection module <NUM>, for online adaptation of unseen (or unknown) animation styles. The additional network can be trained offline in a variety of manners. For example, training data can be based on a labeled dataset that is used for training the detector and dataset of unseen movies (e.g. trailers). The style adaption network can learn to propagate local feature statistics from the dataset used for training to the unseen data. The training can be based on minimax optimization that maximizes the character detector confidence on the characters detected in the unseen images, while minimizing the distance of the deep learned embeddings of the images before and after style transfer (thus maintaining similar semantic information). The deep learned embeddings that can be used are the same that are used for featurizing and grouping.

<FIG> depicts a flow diagram illustrating another example process <NUM> for grouping (or clustering) characters that are automatically detected in a multi-frame animated media file, according to some implementations. The example process <NUM> may be performed in various implementations by a media indexer such as, for example, media indexer <NUM> of <FIG> and <FIG>, or one or more processors, modules, engines, or components associated therewith.

The example process <NUM> is similar to example process <NUM> of <FIG> except that example process <NUM> includes steps for style adaptation. Specifically, the example process <NUM> can adapt or transform features as opposed to entire keyframes (as discussed in the example of <FIG>).

<FIG> illustrates a process <NUM> for sampling negative examples of images to be supplied as training data to an image classifier. Negative examples sampling for image classification provides for classification enhancement. Customizable image classification of any specific domain - for example, animated characters - is required to teach the machine learning model to tell the known classes from the rest of the world. Background sampling is a good way to generate bounding boxes that do not intersect with characters' bounding boxes. The technical problem is the computation complexity since the nature of the problem is a non-convex hard problem in the mathematical sense (NP complete). For instance, the number of possible background (BG) boxes grow exponentially with the number of regions of interests - bounding boxes.

Process <NUM> begins with identifying regions of interest around target content in a frame (step <NUM>). The regions of interest are formed by bounding boxes of rectangular shape drawn around content of interest in an image, series of images (video frames), or the like. Such content includes characters from animated videos, components in a layout (e.g. circuitry and components on an electrical board or furniture in an office layout).

Next, process <NUM> identifies potentially empty regions adjacent to a region of interest (step <NUM>). The region of interest is one of the regions of interest identified in the context of step <NUM>). Each of the potentially adjacent regions include one side that is adjacent to the central region of interest and have axis that are parallel to the axis of the central region of interest. The central region of interest may be a rectangle and the potentially empty regions are also rectangles aligned axially with the central rectangle and having one side that abuts a side of the central rectangle.

Process <NUM> then proceeds to identify at least one empty region, from the potentially empty regions, that satisfies one or more criteria (step <NUM>). An empty region that satisfies the one or more criteria can be classified (or designated) as a negative example of the target content that is the subject of the image classifier (step <NUM>). The negative example can be grouped together with other negative examples in a set and supplied as training data to the classifier, along with positive examples of the target content.

Returning to step <NUM>, identifying empty regions may be accomplished in a variety of ways. In one example, process <NUM> finds the empty region(s) by employing a largest empty rectangle algorithm (Step 1105A). A largest empty rectangle algorithm (or maximum empty rectangle) quickly finds the largest rectangular region in an image that is devoid of the target content. An empty rectangle would therefore avoid - or not overlap with - any other rectangle that includes target content.

Alternatively, process <NUM> may employ a recursive analysis of the regions adjacent to a central region to find the empty regions (step <NUM>). Such a recursive analysis first identifies the regions adjacent to the central region and designates those that are empty (and optionally of a satisfactory size) as negative examples. The analysis then recursively does the same for any of the adjacent regions that are not empty. That is, the analysis identifies other regions that are adjacent to an adjacent region and examines those regions (or sub-regions) for those that are empty (and optionally of a satisfactory size).

<FIG> illustrates the results produced by process <NUM> in an exemplary scenario <NUM>. In this scenario, process <NUM> examples an <NUM> that includes two rectangular bounding boxes represented by bounding box <NUM> and bounding box <NUM>. Bounding box <NUM> is drawn around one animated character, while bounding box <NUM> is drawn around another.

As applied to image <NUM>, process <NUM> utilizing a recursive analysis would identify four empty regions to adjacent to bounding box <NUM> that qualify as negative examples of the target content represented by region <NUM>, region <NUM>, region <NUM>, and region <NUM>. Utilizing a largest rectangle approach, process <NUM> would only identify a single rectangle, e.g. region <NUM> because it is the largest of the four. Whether to use one approach over another would depend on operational constraints. For instance, a largest rectangle approach may be faster than a recursive analysis, but the resulting negative sample may inherently have less information encoded in it than the set of negative examples produced by a recursive approach. However, from a practical perspective, the speed gained by the largest rectangle approach may be considered a worthwhile tradeoff.

<FIG> illustrates a recursive process <NUM> for sampling negative examples of images to be supplied as training data to an image classifier. Aquad-tree like branch and bound recursion is proposed by process <NUM> that yields bounding boxes that are as large as possible under certain complexity constraints. The recursion takes the most centered bounding box and splits the frame four times, i.e. the subframe above, below, on the right and on the left. The stopping criteria is either no more bounding boxes or when the subframe is simply too small. This mechanism allows an indexer and classifier integration to optimize the generation of negative examples even when the image has many bounding boxes which makes the naive approach effectively unsolvable.

Referring more particularly to <FIG>, the process begins with identifying a bounding box in a frame (step <NUM>). The very first box is the center-most box in the frame. The frame includes one or more bounding boxes around animated characters or other objects/regions of interest with respect to which negative examples are needed.

The process proceeds to split the frame into multiple sub-frames around the bounding box (step <NUM>). In one example, four sub-frames could be developed from each of the four sides (left, right, top, and bottom) of the bounding box. Each of the four sub-frames would extend from one side of the bounding box to the edges of the frame itself. In other examples, the bounding box in question may provide fewer than four sides against which to develop sub-frames.

At step <NUM>, the process identifies one of the sub-frames to analyze as potentially acceptable as a negative example and then compares its size to that of a size constraint (step <NUM>). If the size of the sub-frame fails to meet a minimum size (e.g. is less than a threshold size), then the sub-frame is rejected as a potential negative sample. However, if the size of the sub-frame meets the minimum size, then the process determines whether the sub-frame includes or otherwise overlaps with one or more other bounding boxes (step <NUM>).

If no other bounding box is found within the sub-frame, then the sub-frame is considered a negative example and can be categorized or labeled as such (step <NUM>). If, however, the sub-frame includes one or more other bounding boxes within it, then the process returns to step <NUM> to once again split the sub-frame into further sub-frames.

Assuming a sub-frame qualifies as a negative example, the process proceeds to determine if any sub-frames remain with respect to the parent frame to which the sub-frame belongs (step <NUM>). If so, then the process returns to step <NUM> to identify and analyze the next sub-frame.

If no other sub-frames remain, then all the negative examples that were identified can be provided as training data to a classifier (step <NUM>). This step may be performed in batch mode, individually after each negative example is identified, or in some other manner.

<FIG> illustrates an example implementation of the negative example sampling process of <FIG>. In <FIG>, a frame includes two bounding boxes around two characters. The characters are referred to as "red" and "yellow" herein. The larger box <NUM> is drawn around the red character, while the smaller box <NUM> of the two is drawn around the yellow character.

As applied to the image in <FIG>, the negative example sampling process of <FIG> first identifies the center-most bounding box, which is assumed for exemplary purposes to be the larger box around the red character. The frame around the bounding box is divided into four sub-frames to the right <NUM>, left <NUM>, top <NUM>, and bottom <NUM> of the bounding box.

The top sub-frame is determined to satisfy the minimum size criteria and lacks any bounding boxes within it. The top sub-frame therefore qualifies as a negative example. The right sub-frame is also sufficiently large and lacks any other bounding boxes within it and therefore also qualifies as a negative example of the characters. The bottom sub-frame, however, is insufficiently large and therefore is rejected as a negative example candidate.

The left sub-frame, on the other hand, is sufficiently large but includes at least a portion of a bounding box within it - that of the smaller box surrounding the yellow character. The process therefore recursively operates on the portion of the yellow character's bounding box that falls within the left sub-frame of the parent frame.

Like the parent frame, the left sub-frame is split into multiple sub-frames, but only three in this case since the right side of the bounding box around the yellow character is excluded from the left sub-frame. The top sub-frame <NUM> (of the child sub-frame) qualifies as a negative example because it is sufficiently large and has no other bounding boxes within it. The left sub-frame <NUM> (of the child sub-frame) also qualifies as a negative example for the same reasons. However, no sub-frame to the right is possible and the bottom sub-frame <NUM> fails for being too small.

As no other sub-frames exist at either the child level or the parent level of the image frame, all the negative examples have been identified and can be presented to an image classifier to enhance the training thereof. <FIG> illustrates an enlarged view <NUM> of the final four negative examples that were produced by the application of the sampling process of <FIG> to the image of <FIG>.

<FIG> illustrates computing system <NUM> that is representative of any system or collection of systems in which the various processes, programs, services, and scenarios disclosed herein may be implemented. Examples of computing system <NUM> include, but are not limited to, server computers, cloud computing platforms, and data center equipment, as well as any other type of physical or virtual server machine, container, and any variation or combination thereof. Other examples include desktop computers, laptop computers, table computers, Internet of Things (IoT) devices, wearable devices, and any other physical or virtual combination or variation thereof.

Computing system <NUM> may be implemented as a single apparatus, system, or device or may be implemented in a distributed manner as multiple apparatuses, systems, or devices. Computing system <NUM> includes, but is not limited to, processing system <NUM>, storage system <NUM>, software <NUM>, communication interface system <NUM>, and user interface system <NUM> (optional). Processing system <NUM> is operatively coupled with storage system <NUM>, communication interface system <NUM>, and user interface system <NUM>.

Processing system <NUM> loads and executes software <NUM> from storage system <NUM>. Software <NUM> includes and implements process <NUM>, which is representative of the processes discussed with respect to the preceding Figures. When executed by processing system <NUM> to provide packet rerouting, software <NUM> directs processing system <NUM> to operate as described herein for at least the various processes, operational scenarios, and sequences discussed in the foregoing implementations. Computing system <NUM> may optionally include additional devices, features, or functionality not discussed for purposes of brevity.

Continuing with the example of <FIG>, processing system <NUM> may comprise a micro-processor and other circuitry that retrieves and executes software <NUM> from storage system <NUM>. Processing system <NUM> may be implemented within a single processing device but may also be distributed across multiple processing devices or sub-systems that cooperate in executing program instructions. Examples of processing system <NUM> include general purpose central processing units, graphical processing units, application specific processors, and logic devices, as well as any other type of processing device, combinations, or variations thereof.

Storage system <NUM> may comprise any computer readable storage media readable by processing system <NUM> and capable of storing software <NUM>. Storage system <NUM> may include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. Examples of storage media include random access memory, read only memory, magnetic disks, optical disks, flash memory, virtual memory and non-virtual memory, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other suitable storage media. In no case is the computer readable storage media a propagated signal.

Software <NUM> (including learning process <NUM>) may be implemented in program instructions and among other functions may, when executed by processing system <NUM>, direct processing system <NUM> to operate as described with respect to the various operational scenarios, sequences, and processes illustrated herein. For example, software <NUM> may include program instructions for implementing a reinforcement learning process to learn an optimum scheduling policy as described herein.

In particular, the program instructions may include various components or modules that cooperate or otherwise interact to carry out the various processes and operational scenarios described herein. The various components or modules may be embodied in compiled or interpreted instructions, or in some other variation or combination of instructions. The various components or modules may be executed in a synchronous or asynchronous manner, serially or in parallel, in a single threaded environment or multi-threaded, or in accordance with any other suitable execution paradigm, variation, or combination thereof. Software <NUM> may include additional processes, programs, or components, such as operating system software, virtualization software, or other application software. Software <NUM> may also comprise firmware or some other form of machine-readable processing instructions executable by processing system <NUM>.

In general, software <NUM> may, when loaded into processing system <NUM> and executed, transform a suitable apparatus, system, or device (of which computing system <NUM> is representative) overall from a general-purpose computing system into a special-purpose computing system customized to provide motion learning. Indeed, encoding software <NUM> on storage system <NUM> may transform the physical structure of storage system <NUM>. The specific transformation of the physical structure may depend on various factors in different implementations of this description. Examples of such factors may include, but are not limited to, the technology used to implement the storage media of storage system <NUM> and whether the computer-storage media are characterized as primary or secondary storage, as well as other factors.

For example, if the computer readable storage media are implemented as semiconductor-based memory, software <NUM> may transform the physical state of the semiconductor memory when the program instructions are encoded therein, such as by transforming the state of transistors, capacitors, or other discrete circuit elements constituting the semiconductor memory. A similar transformation may occur with respect to magnetic or optical media. Other transformations of physical media are possible without departing from the scope of the present description, with the foregoing examples provided only to facilitate the present discussion.

Communication interface system <NUM> may include communication connections and devices that allow for communication with other computing systems (not shown) over communication networks (not shown). Examples of connections and devices that together allow for inter-system communication may include network interface cards, antennas, power amplifiers, RF circuitry, transceivers, and other communication circuitry. The connections and devices may communicate over communication media to exchange communications with other computing systems or networks of systems, such as metal, glass, air, or any other suitable communication media. The aforementioned communication networks and protocols are well known and need not be discussed at length here. However, some communication protocols that may be used include, but are not limited to, the Internet protocol (IP, IPv4, IPv6, etc.), the transfer control protocol (TCP), and the user datagram protocol (UDP), as well as any other suitable communication protocol, variation, or combination thereof.

Communication between computing system <NUM> and other computing systems (not shown), may occur over a communication network or networks and in accordance with various communication protocols, combinations of protocols, or variations thereof. Examples include intranets, internets, the Internet, local area networks, wide area networks, wireless networks, wired networks, virtual networks, software defined networks, data center buses and backplanes, or any other type of network, combination of network, or variation thereof. The aforementioned communication networks and protocols are well known and need not be discussed at length here.

The phrases "in some embodiments," "according to some embodiments," "in the embodiment shown," "in other embodiments," "in some implementations," "according to some implementations," "in the implementation shown," "in other implementations," and the like generally mean the particular feature, structure, or characteristic following the phrase is included in at least one embodiment or implementation of the present technology and may be included in more than one embodiment or implementation. In addition, such phrases do not necessarily refer to the same or different embodiments or implementations.

The functional block diagrams, operational scenarios and sequences, and flow diagrams provided in the Figures are representative of exemplary systems, environments, and methodologies for performing novel aspects of the disclosure. While, for purposes of simplicity of explanation, methods included herein may be in the form of a functional diagram, operational scenario or sequence, or flow diagram, and may be described as a series of acts, it is to be understood and appreciated that the methods are not limited by the order of acts, as some acts may, in accordance therewith, occur in a different order and/or concurrently with other acts from that shown and described herein. For example, those skilled in the art will understand and appreciate that a method could alternatively be represented as a series of interrelated states or events, such as in a state diagram. Moreover, not all acts illustrated in a methodology may be required for a novel implementation.

Claim 1:
A method for indexing video comprising:
identifying one or more regions of interest (<NUM>, <NUM>) around target content in a frame (<NUM>) of the video, wherein the target content comprises an animated character in the video;
identifying, in a portion of the frame outside the one or more regions of interest, potentially empty regions (<NUM>, <NUM>, <NUM>, <NUM>) adjacent to the one or more regions of interest, wherein the one or more regions of interest comprises a bounding box around the animated character, wherein the potentially empty regions adjacent to the one or more regions of interest comprise rectangles, each with one side adjacent to the bounding box and wherein the rectangles comprise empty rectangles that do not overlap with any of the one or more regions of interest around the target content;
identifying at least one empty region of the potentially empty regions that satisfies one or more criteria, said identifying comprising identifying a largest one of the empty rectangles, wherein the at least one empty region is devoid of the target content; and
classifying at least the one empty region as a negative sample of the target content.