Adaptive action detection

Described is providing an action model (classifier) for automatically detecting actions in video clips, in which unlabeled data of a target dataset is used to adaptively train the action model based upon similar actions in a labeled source dataset. The target dataset comprising unlabeled video data is processed into a background model. The action model is generated from the background model using a source dataset comprising labeled data for an action of interest. The action model is iteratively refined, generally by fixing a current instance of the action model and using the current instance of the action model to search for a set of detected regions (subvolumes), and then fixing the set of subvolumes and updating the current instance of the action model based upon the set of subvolumes, and so on, for a plurality of iterations.

BACKGROUND

Automated recognition of human actions in video clips has many useful applications, including surveillance, health care, human computer interaction, computer games, and telepresence. In general, a trained action classifier (model) processes the video clips to determine whether a particular action takes place.

To learn an effective action classifier model, previous approaches rely on a significant amount of labeled training data, i.e., training labels. In general, this works well for one dataset, but not another. For example, the background, lighting, and so forth may be different across datasets.

As a result, to recognize the actions in a different dataset, heretofore labeled training data approaches have been used to retrain the model, using new labels. However, labeling video sequences is a very tedious and time-consuming task, especially when detailed spatial locations and time durations are needed. For example, when the background is cluttered and there are multiple people appearing in the same frame, the labelers need to provide a bounding box for every subject together with the starting/ending frames of an action instance. For a video as long as several hours, the labeling process may take on the order of weeks.

SUMMARY

Briefly, various aspects of the subject matter described herein are directed towards a technology by which unlabeled data of a target dataset is used to adaptively train an action model based upon similar actions in a source dataset. In one aspect, the target dataset containing (e.g., unlabeled) video data is processed into a background model. The background model is processed into an action model by using a source dataset with video data (e.g., labeled) that includes action of interest data. The target dataset is searched to find one or more detected regions (subvolumes) where similar action of interest data occurs. When found, the action model is updated based on the detected regions. The action model is refined by iteratively searching with the most current action model, then updating that action model based upon the action of interest regions, and so forth, for a number of iterations. The action model may then be output for use as a classifier.

In one aspect, the background model and action model comprise spatial-temporal interest points modeled as Gaussian mixture models. Searching the target dataset comprises performing a localization task based on scoring a function, in which the searching may comprise branch and bound searching.

DETAILED DESCRIPTION

Various aspects of the technology described herein are generally directed towards an adaptive action detection approach that combines model adaptation and action detection. To this end, the technology trains a background model from unlabeled data (in a target dataset), and uses this background model and labeled training data (in a source dataset) to extract a foreground (action of interest) model (a classifier). The action of interest model may be iteratively refined via further data processing. In this way, the technology effectively leverages unlabeled target data in adapting the action of interest model in a cross-dataset action detection approach. As can be readily appreciated, such cross-dataset action detection is valuable in many scenarios, such as surveillance applications.

It should be understood that any of the examples herein are non-limiting. As such, the present invention is not limited to any particular embodiments, aspects, concepts, structures, functionalities or examples described herein. Rather, any of the embodiments, aspects, concepts, structures, functionalities or examples described herein are non-limiting, and the present invention may be used in various ways that provide benefits and advantages in computing and video processing in general.

FIG. 1is a block diagram showing an adaptive action detection mechanism102that reduces the need for training labels/handles the task of cross-dataset action detection with few or no extra training labels. In general, cross-dataset action detection aims to generalize action detection models present in a source dataset104to a target dataset106.

The mechanism102processes the source dataset104, such as containing labeled data in a relatively simpler dataset, e.g., containing only video clips recorded with clean backgrounds, with each video clip involving only one type of repetitive action of a single person. In contrast, in another dataset (e.g., a target dataset106), the background may be cluttered, and there may be multiple people moving around with occasional occlusions. As described herein, based on the source dataset104, a background model108and an iterative procedure, the adaptive action detection mechanism102processes the target dataset106to recognize similar actions therein, and iteratively repeats the processing to generate and refine an action of interest model110for action classification.

As represented inFIG. 1by the arrow labeled with circled numeral one (1), adaptive action detection mechanism102extracts STIPs (spatial-temporal interest points) from video clips and estimates the distribution of a background model108. More particularly, histogram of oriented gradients optical flow features are extracted at salient locations in a known manner, with a high feature dimension. Note that STIPs in videos provide rich representations of local gradients of edges and motions, however such features are often affected by lighting conditions, viewpoints, and partial occlusions, whereby STIPs from different datasets may have different distributions. To overcome this problem, instead of using STIPs based on quantized histograms, a probabilistic representation of the original STIPs is employed, by applying PCA (principal component analysis) to the STIPs, and then modeling them with (e.g., 512-component) Gaussian Mixture Models (GMMs). GMMs with large number of components are known to be able to model any given probability distribution function. To model the correlation between a source dataset and a target dataset, a prior distribution of the GMM parameters is used, in conjunction with an adaptation approach to incorporate the prior information for cross-dataset analysis.

As represented inFIG. 1by the arrows labeled two (2), the adaptive action detection mechanism102adapts the background model108to an action of interest model110using the labeled data in the source dataset104. In general, the background model108contains both background and actions of interest, and the source dataset104is used to extract similar action data (positive data) from the background model108. If the source dataset is labeled, then the labels may be transferred to the appropriate actions in the target dataset, however even if unlabeled, similar actions may be detected. Details of the adaptation are described below.

With the adapted action of interest model110, the adaptive action detection mechanism102estimates the location of an action instance (block112) in the target dataset106by differentiating between the STIPs in the background and the STIPs for an action of interest. This is generally represented inFIG. 1by the arrows labeled three (3), which are shown as dashed lines to help distinguish them from the other arrows. Note that the operations indicated by the arrows labeled two (2) and three (3) are iteratively repeated, as described below.

Turning to additional details, one example approach combines model adaptation and action detection into a Maximum a Posterior (MAP) estimation framework, which explores the spatial-temporal coherence of actions and good use of the prior information that can be obtained without supervision. The technology described herein combines action detection and classifier adaptation into a single framework, which provides benefits in cross-dataset detection. Note that cross-dataset learning and semi-supervised learning learn a model with limited amount of labeled data. However, semi-supervised learning assumes the labeled data and unlabeled data are generated from the same distribution, which is not the case in cross-dataset learning. In cross-dataset learning, the actions of interest are assumed to share some similarities across the datasets, but are not exactly the same, and further, the background models are typically significantly quite different from dataset to dataset. Because of this, semi-supervised learning is not suitable, and the technology described herein instead treats the action model in the source dataset as prior, and employs maximum a posterior estimation to adapt the model to the target dataset. Moreover, the model provides both classification and detection (spatial and temporal localization), while conventional semi-supervised learning algorithms considers only classification.

Another aspect is directed towards the spatial-temporal coherence nature of the video actions, using a three-dimensional (3D) subvolume to represent a region in the 3D video space that contains an action instance. As generally described in U.S. patent application Ser. No. 12/481,579, herein incorporated by reference, a 3D subvolume is parameterized as a 3D cube with six degrees of freedom in (x, y, t) space. Spatial and temporal localization of an action in a video sequence is rendered as searching for the optimal subvolume.

The technology described herein simultaneously locates the action and updates the GMM parameters. The benefits of doing so include that action detection provides more useful information to the user, and that locating the spatial-temporal subvolumes allows iteratively filtering out the STIPs in the background, thus refining the model estimation.

A video sequence may be represented as a collection of spatial-temporal interests points (STIPs), where each STIP is represented by a feature vector q. To model the probability of each STIP, a Gaussian Mixture Model represents a universal background distribution. If a GMM contains K components, the probability can be written as

Pr⁡(q|θ)=∑k=1K⁢wk⁢𝒩⁡(q;μk,Σk)
where(•) denotes the normal distribution, and μkand Σkdenote the mean and variance of the kth normal component, respectively. Each component is associated with a weight wkthat satisfies Σk=1Kwk=1. The parameter of the GMM is denoted by θ={μk, Σk, wk}, of which the prior takes the form:

Pr⁡(θ)=∏k⁢⁢Pr⁡(μk,Σk)(1)
where Pr(μk, Σk) is a normal-Wishart density distribution representing the prior information. In cross-dataset detection, Pr(θ) represents the prior information obtained from the source dataset104. It is generally likely that in different datasets the action information may be correlated, and thus the prior Pr(θ) from the source dataset is likely beneficial for the action detection in the target dataset106.

The task of action detection needs to distinguish the action of interest from the background. To this end, two GMM models are employed, namely a background model108, θb={μbk, Σbk, wbk}, and the model110for the actions of interest, θc={μck, Σck, wck}. The corresponding prior distributions are denoted as Pr(θb) and Pr(θc), respectively. The task of action detection is modeled as finding 3D subvolumes in spatial and temporal domains that contain the actions of interest.

Let Q={Q1, Q2, . . . } denote the set of subvolumes, each of which contain an instance of the action. The union of Q is UQ=UQεQQ and letUQdenote the complement of UQ. By assuming that each STIP is independent of one another, the log likelihood of STIPs can be written as:

To detect action in the target dataset, the mechanism102needs to find the optimal action model θctogether with the action subvolume Q in the target dataset106.

However, directly optimizing equation (3) is intractable. An effective approach in practice is to find the solution in an iterative way:

Equation (4) provides a tool to incorporate the cross-dataset information. By way of example, consider a labeled source dataset S and an unlabeled target dataset T. The process can estimate θbby fitting the GMM with the STIPs in T. However, it is difficult to estimate θcbecause there is no label information of Q in T. However, θcmay be obtained by applying equation (4) to the source dataset S. With an initial θc, the label information in S may be used to update the estimation of θcand Q in T. This approach adapts the action model from S to T and is referred to as adaptive action detection. The following algorithm, also represented inFIG. 2, describes the adaptive action detection process.

Iterative Adaptive Action Detection

Step 201:Input the labeled source dataset S and target dataset T.Step 202:Train background model Pr(θb) based on the STIPs in the T.In Source dataset S:Step 203:apply equation (4) to S and obtain θc.In Target dataset T:Step 204:update equation Q using equation (5),Step 205:update equation θcusing equation (4),Step 206:repeat steps 204 and 205 for several rounds.Step 207:Output the action model and the detected regions in T.

In general, the objective function is optimized by fixing the action model and updating the set of subvolumes Q containing the action, then fixing the set of subvolumes and optimizing the action model θc, and so on until convergence is detected (e.g., if the measured improvement between iterations is below some threshold) or some fixed number of iterations is performed. The action model may then be output for use as a classifier, along with data corresponding to the subvolumes (the detected regions of action of interest), e.g., the frames and locations therein.

Turning to computing the updated action model θc*, the optimal parameter θcmaximizes equation (2):

When Q is given and the background model θbis fixed, the problem is simplified as:

The model of μckin the source dataset is taken as prior. Because Gaussian distribution is the conjugate prior for Gaussian, the MAP estimation for equation (6) is obtained in a simple form:
μck=αkEck(x)+(1−αk)μck
Σck=βkEck(x2)+(1−βk)(Σck+μckTμck)−μckTμck(7)
where αkrepresents the weights that adjust the contribution of the prior model to the updated model. The variable Eckis the weighted summation of samples in the target dataset. Note that for faster speed and robustness, only μk(and not Σk) is updated. The variable Eckcan be estimated as:

Eck=1∑j⁢pkj⁢∑qj∈UQ⁢pkj⁢qj⁢⁢pkj=wk⁢𝒩⁡(qj|μk,Σk)∑k⁢wk⁢𝒩⁡(qj|μk,Σk)(8)
Note that the weighting parameter αkalso may be simplified as:

αk=∑j⁢pkj∑j⁢pkj+r
where r is the controlling variable for adaptation. The adaptation approach effectively makes use of the prior information from source dataset104, and requires only a small amount of training data to obtain the adaptation model.

Turning to subvolume detection, given the model θc, the best subvolume containing the action of interest can be found.

The second term is constant given the universal background model108. Thus, a simplified form of subvolume detection may be used:

UQ*=arg⁢maxUQ⁢∑q∈UQ⁢log⁢Pr⁡(q|θc)⁢Pr⁡(θc)Pr⁡(q|θb)⁢Pr⁡(θb)(10)
Assigning each STIP a score:

f⁡(UQ)=∑q∈UQ⁢f⁡(q).(11)
It is known that the localization task based on the scoring function in (11) can be accomplished efficiently by a well-known Branch and Bound (BB) search method. Denoting UQas the collection of 3D subvolumes gives Q ε UQ. Assuming that there are two subvolumes Qminand Qmax, such that Qmin⊂Q⊂Qmax, let ƒ+and ƒ−be two functions defined as:

f+⁡(Q)=∑q∈Q⁢max⁡(f⁡(q),0)f-⁡(Q)=∑q∈Q⁢min⁡(f⁡(q),0),
which gives:
ƒ(Q)≦ƒ0(Q)=ƒ+(Qmax)+ƒ−(Qmin)  (12)
for every Q ε UQ, which is the basic upper bound function used in BB search.

In this manner, there is provided cross-dataset learning that adapts an existing classifier from a source dataset to a target dataset, while using only a small amount of labeling samples (or possibly no labels at all). The cross-dataset action detection is possible even though videos may be taken on different occasions, with different backgrounds, with actions that may appear differently with different people, and with different lighting conditions, scales and action speeds, and so forth. Notwithstanding, the actions in different datasets still share some similarities to an extent, and the classifier adaptation leverages the spatial and temporal coherence of the individual actions in the target dataset.

EXEMPLARY OPERATING ENVIRONMENT

FIG. 3illustrates an example of a suitable computing and networking environment300on which the examples ofFIGS. 1 and 2may be implemented. The computing system environment300is only one example of a suitable computing environment and is not intended to suggest any limitation as to the scope of use or functionality of the invention. Neither should the computing environment300be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the exemplary operating environment300.

With reference toFIG. 3, an exemplary system for implementing various aspects of the invention may include a general purpose computing device in the form of a computer310. Components of the computer310may include, but are not limited to, a processing unit320, a system memory330, and a system bus321that couples various system components including the system memory to the processing unit320. The system bus321may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. By way of example, and not limitation, such architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus also known as Mezzanine bus.

The system memory330includes computer storage media in the form of volatile and/or nonvolatile memory such as read only memory (ROM)331and random access memory (RAM)332. A basic input/output system333(BIOS), containing the basic routines that help to transfer information between elements within computer310, such as during start-up, is typically stored in ROM331. RAM332typically contains data and/or program modules that are immediately accessible to and/or presently being operated on by processing unit320. By way of example, and not limitation,FIG. 3illustrates operating system334, application programs335, other program modules336and program data337.

The computer310may also include other removable/non-removable, volatile/nonvolatile computer storage media. By way of example only,FIG. 3illustrates a hard disk drive341that reads from or writes to non-removable, nonvolatile magnetic media, a magnetic disk drive351that reads from or writes to a removable, nonvolatile magnetic disk352, and an optical disk drive355that reads from or writes to a removable, nonvolatile optical disk356such as a CD ROM or other optical media. Other removable/non-removable, volatile/nonvolatile computer storage media that can be used in the exemplary operating environment include, but are not limited to, magnetic tape cassettes, flash memory cards, digital versatile disks, digital video tape, solid state RAM, solid state ROM, and the like. The hard disk drive341is typically connected to the system bus321through a non-removable memory interface such as interface340, and magnetic disk drive351and optical disk drive355are typically connected to the system bus321by a removable memory interface, such as interface350.

The drives and their associated computer storage media, described above and illustrated inFIG. 3, provide storage of computer-readable instructions, data structures, program modules and other data for the computer310. InFIG. 3, for example, hard disk drive341is illustrated as storing operating system344, application programs345, other program modules346and program data347. Note that these components can either be the same as or different from operating system334, application programs335, other program modules336, and program data337. Operating system344, application programs345, other program modules346, and program data347are given different numbers herein to illustrate that, at a minimum, they are different copies. A user may enter commands and information into the computer310through input devices such as a tablet, or electronic digitizer,364, a microphone363, a keyboard362and pointing device361, commonly referred to as mouse, trackball or touch pad. Other input devices not shown inFIG. 3may include a joystick, game pad, satellite dish, scanner, or the like. These and other input devices are often connected to the processing unit320through a user input interface360that is coupled to the system bus, but may be connected by other interface and bus structures, such as a parallel port, game port or a universal serial bus (USB). A monitor391or other type of display device is also connected to the system bus321via an interface, such as a video interface390. The monitor391may also be integrated with a touch-screen panel or the like. Note that the monitor and/or touch screen panel can be physically coupled to a housing in which the computing device310is incorporated, such as in a tablet-type personal computer. In addition, computers such as the computing device310may also include other peripheral output devices such as speakers395and printer396, which may be connected through an output peripheral interface394or the like.

The computer310may operate in a networked environment using logical connections to one or more remote computers, such as a remote computer380. The remote computer380may be a personal computer, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to the computer310, although only a memory storage device381has been illustrated inFIG. 3. The logical connections depicted inFIG. 3include one or more local area networks (LAN)371and one or more wide area networks (WAN)373, but may also include other networks. Such networking environments are commonplace in offices, enterprise-wide computer networks, intranets and the Internet.

When used in a LAN networking environment, the computer310is connected to the LAN371through a network interface or adapter370. When used in a WAN networking environment, the computer310typically includes a modem372or other means for establishing communications over the WAN373, such as the Internet. The modem372, which may be internal or external, may be connected to the system bus321via the user input interface360or other appropriate mechanism. A wireless networking component such as comprising an interface and antenna may be coupled through a suitable device such as an access point or peer computer to a WAN or LAN. In a networked environment, program modules depicted relative to the computer310, or portions thereof, may be stored in the remote memory storage device. By way of example, and not limitation,FIG. 3illustrates remote application programs385as residing on memory device381. It may be appreciated that the network connections shown are exemplary and other means of establishing a communications link between the computers may be used.

An auxiliary subsystem399(e.g., for auxiliary display of content) may be connected via the user interface360to allow data such as program content, system status and event notifications to be provided to the user, even if the main portions of the computer system are in a low power state. The auxiliary subsystem399may be connected to the modem372and/or network interface370to allow communication between these systems while the main processing unit320is in a low power state.

CONCLUSION