Substructure and boundary modeling for continuous action recognition

Embodiments of the present invention include systems and methods for improved state space modeling (SSM) comprising two added layers to model the substructure transition dynamics and action duration distribution. In embodiments, the first layer represents a substructure transition model that encodes the sparse and global temporal transition probability. In embodiments, the second layer models the action boundary characteristics by injecting discriminative information into a logistic duration model such that transition boundaries between successive actions can be located more accurately; thus, the second layer exploits discriminative information to discover action boundaries adaptively.

TECHNICAL FIELD

The present patent document is directed towards systems and methods for segmenting and recognizing of actions.

DESCRIPTION OF THE RELATED ART

Vision-based action recognition has wide application. For example, vision-based action recognition may be used in driving safety, security, signage, home care, robot training, and other applications.

One important application of vision-based action recognition is Programming-by-Demonstration (PbD) for robot training. For Programming-by-Demonstration, a task to train is often decomposed into primitive action units. For example, in Programming-by-Demonstration, a human demonstrates a task that is desired to be repeated by a robot. While the human demonstrates the task, the demonstration process is captured by sensors, such as a video or videos using one or more cameras. These videos are segmented into individual unit actions, and the action type is recognized for each segment. The recognized actions are then translated into robotic operations for robot training.

Understanding continuous activities from videos using, for example, simultaneous segmentation and classification of actions, is a fundamental yet challenging problem in computer vision. Many existing works approach the problem using bottom-up methods, where segmentation is performed as preprocessing to partition videos into coherent constituent parts, and action recognition is then applied as an isolated classification step. Although literature exists for segmentation of time series, such as change point detection, periodicity of cyclic events modeling, and frame clustering, the methods tend to detect local boundaries and lack the ability to incorporate global dynamics of temporal events, which leads to under or over segmentation that severely affects the recognition performance, especially for complex actions with diversified local motion statistics.

The limitation of the bottom-up approaches has been addressed by performing concurrent top-down recognition using variants of Dynamic Bayesian Network (DBN), where the dynamics of temporal events are modeled as transitions in a latent or partially observed state space. The technique has been used in speech recognition and natural language processing, while the performance of existing DBN-based approaches for action recognition tends to be relatively lower, mostly due to the difficulty in interpreting the physical meaning of latent states. Thus, it becomes difficult to impose additional prior knowledge with clear physical meaning into an existing graphical structure to further improve its performance.

Accordingly, what is needed are improved systems and methods for segmentation and classification of actions.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Also, it shall be noted that steps or operations may be performed in different orders or concurrently, as will be apparent to one of skill in the art. And, in instances, well known process operations have not been described in detail to avoid unnecessarily obscuring the present invention.

Furthermore, connections between components within the figures are not intended to be limited to direct connections. Rather, data between these components may be modified, re-formatted, or otherwise changed by intermediary components. Also, additional or fewer connections may be used. It shall also be noted that the terms “coupled” or “communicatively coupled” shall be understood to include direct connections, indirect connections through one or more intermediary devices, and wireless connections.

Reference in the specification to “one embodiment,” “preferred embodiment,” “an embodiment,” or “embodiments” means that a particular feature, structure, characteristic, or function described in connection with the embodiment is included in at least one embodiment of the invention and may be in more than one embodiment. The appearances of the phrases “in one embodiment,” “in an embodiment,” or “in embodiments” in various places in the specification are not necessarily all referring to the same embodiment or embodiments.

The use of certain terms in various places in the specification is for illustration and should not be construed as limiting. A service, function, or resource is not limited to a single service, function, or resource; usage of these terms may refer to a grouping of related services, functions, or resources, which may be distributed or aggregated.

Embodiments of the present invention presented herein will be described using video data and human action examples. These examples are provided by way of illustration and not by way of limitation. One skilled in the art shall also recognize the general applicability of the present inventions to other applications.

A. General Overview

Due to the ineffectual results of prior approaches, the present patent document presents how at least two additional sources of information with clear physical interpretations can be considered in a general graphical structure for State-Space Model (SSM). Compared to a standard Switching Linear Dynamic System (SLDS)100, shown inFIG. 1(a), where X, Y, and S are respectively the hidden state, the observation, and the label, the new model105inFIG. 1(b) is augmented with two additional components or nodes, Z and D, to consider the substructure transition and duration statistics of actions.FIG. 1(b) depicts an embodiment of the structure of model105according to embodiments of the present invention, in which an action is represented by an SLDS with substructure transition, and the inter-action transition is controlled by discriminative boundary model.

Rather than a uniform motion type, a real-world human action is usually characterized by a set of inhomogeneous units with some instinct structure, which may be referred to herein as “substructure.” Action substructure typically arises from two factors: (1) the hierarchical nature of activity, where one action can be temporally decomposed into a series of primitives with spatial-temporal constraints; (2) the large variance of action dynamics due to differences in kinematical property of subjects, feedback from environment, or interaction with objects.

For the first factor, multi-class Support Vector Machine (SVM) with Dynamic Programming has been used to recognize coherent motion constituent parts in an action. Latent-SVM has been applied for temporal evolving of “attributes” in actions. Also, a two-layer Maximum Entropy Markov Models (MEMM) has been suggested to recognize the correspondence between sub-activities and human skeletal features.

For the second factor, methods have been proposed to consider the substructure variance caused by subject-object interaction with Connected Hierarchic Conditional Random Field (CRF), and the substructure caused by pose variance with Latent Pose CRF.

In more general case, a Latent Dynamic CRF (LDCRF) algorithm has been presented. The LDCRF algorithm includes an added “latent-dynamic” layer into CRF for hidden substructure transition. One key limitation with CRF as a discriminative method is that one single pseudo-likelihood score is estimated for an entire sequence, which is incapable to interpret the probability of each individual frame. To solve the problem, embodiments presented herein include a generative model105as presented inFIG. 1(b), with extra hidden node Z gating the transition amongst a set of dynamic systems, and the posterior for every action can be inferred strictly under Bayesian framework for each individual frame. Since generative model usually requires large amount of training data, to overcome the limitation, effective prior constraints are introduced in the training process as explained in Section B (below).

2. Duration Model

The duration information of actions is helpful in determining the boundary where one action transits to another in continuous recognition tasks. Duration model has been adopted in Hidden Markov Model (HMM) based methods, such as the explicit duration HMM or more generally the Hidden Semi Markov Model (HSMM). Incorporating duration model into SSM is more challenging than HMM because SSM has continuous state space, and exact inference in SSM is intractable. Some works reported in this line include for music transcription and for economics. A duration constraint has been imposed at the top level of SLDS to achieve improved performance for honeybee behavior analysis. In general, naive integration of duration model into SSM is not effective, because duration patterns vary significantly across visual data and limited training samples may bias the model with incorrect duration patterns.

To address this problem, as presented in the embodiment depicted inFIG. 1(b), the model105correlates duration node D with the continuous hidden state node X and the substructure transition node Z as explained in Section C. In this way, the duration model becomes more discriminative than conventional generative duration models, and the data-driven boundary locating process can accommodate more variation in duration length.

In summary, aspects of the present invention incorporate at least two additional models into a general SSM, namely the Substructure Transition Model (STM) and the Discriminative Boundary Model (DBM). In embodiments, a Rao-Blackwellised particle filter is also designed for efficient inference of the model in Section D. Embodiments of efficient training and inference algorithms to support continuous action recognition are also presented.

Experiments in Section F demonstrate the superior performance of embodiments of the present invention over several existing state-of-the-arts in continuous action recognition.

Linear Dynamic Systems (LDS) is the most commonly used SSM to describe visual features of human motions. LDS is modeled by the following distributions:
p(Yt=yt|Xt=xt)=(yt;Bxt,R)  (1)
p(Xt+1=xt+1|Xt=xt)=(xt+1;Axt,Q)  (2)

where Ytis the observation at time t, Xtis a latent state,(x; μ, Σ) is multivariate normal distribution of x with mean μ and covariance Σ. To consider multiple actions, SLDS is formulated as a mixture of LDS's with the switching among them controlled by action class St. However, each LDS can only model an action with homogenous motion, ignoring the complex substructure within the action. In embodiments, a discrete hidden variable Ztis introduced to explicitly represent such information, and the substructure transition model can be stated as:
p(Yt=yt|Xt=xt,Sti,Ztj)=(yt;Bijxt,Rij)  (3)
p(Xt+1=xt+1|Xt=xt,Sti,Zt+1j)=(xt+1;Aijxt,Qij)  (4)

where Aij, Bij, Qij, and Rijare the LDS parameters for the jthaction primitive in the substructure of ithaction class. In embodiments, {Zt} is modeled as a Markov chain and the transition probability is specified by multinomial distribution:
p(Zt+1j|Zti,Stk)=θijk(5)

In the following, the term STM may refer to either the transition matrix in Equation (5) or the overall substructured SSM depending on its context. Some examples of STM are given inFIG. 2, which presents STM trained for action “move-arm” in a stacking dataset using (a) sparse (200) and (b) block-wise sparse constraints (205), with NZ=5 and NQ=3. Note that the STM205inFIG. 2better captures global ordering. The STM training is explained in more detail in the remainder of this section.

In embodiments, a simplified notation Θ={θij} will be used for the STM within a single action. An unconstrained Θ implies that the substructure of action primitives may be organized in an arbitrary way. In embodiments, for most real-world human actions, however, there is a strong temporal ordering associated with the primitive units. Such order relationship can be vital to accurate action recognition as well as robust model estimation.

There have been some attempts to encode a fixed order relationship among primitive units by restricting the locations of non-zero elements in transition matrix Θ; examples include the left-to-right HMM, switching HMM (SHMM), and factorial HMM.FIG. 3depicts, by way of example, the difference between prior approaches and that of the present patent document according to embodiments of the present invention. Unconstrained transition matrices, such as those in HMM and SLDS305, are deficient because they fail to appreciate strong temporal ordering that can exist with the primitive units. Others that apply some constraints, such as left-to-right HMM310, specify the temporal ordering a priori. In many cases, it is difficult to specify the temporal ordering a priori, and a more practical approach is to impose a sparse transition constraint while leaving the discovery of exact order relationship to training phase. As shown inFIG. 3, the dynamic substructure modeling facilitates only some blocks in the transition matrix315taking positive value. This enables the modeling of sequential structure as well as local variation in action sequences (e.g.,320-A,320-B,320-C).

Along this direction, negative Dirichlet distribution has been proposed as a prior for each row θiin Θ:

where α is a pseudo count penalty. The maximum a posteriori probability (MAP) estimation of parameter is

where ξijis the sufficient statistics ofZti, Zt+1j. When the number of transitions from zito zjin training data is less than α, the probability θijis set to zero. The sparsity enforced in this way often leads to local transition patterns sensitive to noise or incomplete data, as show inFIG. 2(a). Also, the penalty term α introduces bias to the proportion of non-zero transition probabilities, i.e.

θ^ijθ^ik≠ξijξik.
In embodiments, this bias may be severe especially when ξijis small.

In embodiments, for tradeoff between model sparsity and flexibility, a block-wise sparse transition model may be used to regularize the global topology of action substructure. The idea is to divide an action into several stages and each stage comprises a subset of action units. In embodiments, the transition between stages is encouraged to be sequential but sparse, such that the global action structure can be modeled. At the same time, the action units within each stage can propagate freely from one to another so that variation in action styles and parameters is also preserved.

In embodiments, formally, define discrete variable Qtε{1, . . . , NQ} as the current stage index of action, and assume a surjective mapping function g(•) is given which assigns each action primitive Zt to its corresponding stage Qt:

The choice of g(•) depends on the nature of action. Intuitively, more action units may be assigned to a stage with diversified motion patterns and less action units to a stage with restricted pattern. In embodiments, the joint dynamic transition distribution of Qtand Ztis:
p(Qt+1,Zt+1|Qt,Zt)=p(Qt+1|Qt)p(Zt+1|Qt+1,Zt)  (8)

In embodiments, the second term of Equation (8) specifies the transition between action primitives, which are kept flexible to model diversified local action patterns. The first term captures the global structure between different action stages, and therefore, in embodiments, an ordered negative Dirichlet distribution is imposed as its hyper-prior:

where Φ={φqr} is the stage transition probability matrix, φqr=p(Qt+1r|Qtq), and α is a constant for pseudo count penalty. The ordered negative Dirichlet prior encodes both sequential order information and sparsity constraint. It promotes statistically a global transition path Q1→Q2→ . . . →QNQwhich can be learned from training data rather than heuristically defined as that in left-to-right HMM. An example of a resulting STM is shown inFIG. 2(b)205, where action unit Ztcan transit, with certain probability, from the starting stage Q1to the intermediate stage Q2, or from Q2to the terminating stage Q3, while transiting directly from Q1to Q3is prohibited. Note that, in embodiments, no in-coming/out-going transition is encouraged for Q1/QNQ, which stands for starting/terminating stage. In embodiments, the identification of these two special stages is helpful for segmenting continuous actions, as will be discussed in Section C.2.

In embodiments, the MAP model estimation involves maximizing the product of likelihood (Equation (8)) and prior (Equation (9)) under the constraint of Equation (7). There are two interdependent nodes, Q and Z, involved in the optimization, which make the problem complicated. Equation (8) may be replaced with the transition distribution of single variable Z in Equation (5), and a constraint exists for the relationship between Θ and Φ. Therefore, in embodiments, the node Q (and the associated parameter Φ) serves for conceptual purpose and may be eliminated in final model construction. In embodiments, the MAP estimation can be converted to the following constrained optimization problem:

G⁡(q)⁢=Δ⁢{i|g⁡(i)=q},
and {φqr} are auxiliary variables. In embodiments, the optimal solution is

where αqris equal to α if q≠r or (q+1)≠r, and 0 otherwise. As can be seen, the resultant {circumflex over (Θ)} is a block-wise sparse matrix, which can characterize both the global structure and local detail of action dynamics. Also, within each block (stage), there is no bias in {circumflex over (θ)}ij.

For one of ordinary skill in the art, it is straightforward to use a Markov chain to model the transition of action Stby, p(St+1j|Sti)=aij. The duration information of the ithaction is naively incorporated into its self-transition probability aii, which leads to an exponentially-distributed action duration model:
p(duri=τ)=aiiτ−1(1−aii), τ=1, 2, 3 . . .

Unfortunately, only a limited number of real-life events have an exponentially diminishing duration. Inaccurate duration modeling can severely affect ability to segment consecutive actions and identify their boundaries.

In embodiments, non-exponential duration distribution may be implemented with duration-dependent transition matrix, such as the one used in HSMM. Fitting a transition matrix for each epoch in maximum length of duration is often impossible given a limited number of training sequences, even parameter hyperprior such as hierarchical Dirichlet distribution is used to restrict model freedom. Parametric duration distributions such as gamma and Gaussian provide a more compact way to represent duration and show good performance in signal synthesis. However, they are less useful in inference because the corresponding transition probability is not easy to evaluate.

1. Logistic Duration Model

In embodiments, a new logistic duration model is provided to overcome the above limitation. In embodiments, a variable Dtis introduced to represent the length of time current action has been lasting. {Dt} is a counting process starting from 1, and the beginning of a new action is triggered whenever it is reset to 1:

where aijis the probability of transiting from previous action i to new action j. Notice that the same type of action may be repeated if aii>0.

In embodiments, instead of modeling action duration distribution directly, we model the transition distribution of Dtas a logistic function of its previous value:

where νiand βiare positive logistic regression weights. Equation (13) immediately leads to the duration distribution for action class i:

FIG. 4(a) shows how the probability of Dt+1changes as a function of Dtwith different parameter sets, and the corresponding action duration distribution is plotted inFIG. 4(b). The increasing probability of transiting to a new action leads to a duration distribution, with center and width controlled by βiand νi, respectively.

In embodiments, merely stacking the logistic duration layer (D-S) onto the STM layer (Z-X-Y) leads to a generative SSM, which is unable to utilize contextual information for accurate action boundary segmentation. Discriminative graphic models, such as MEMM and CRF, are generally more powerful in such classification problem except that they ignore data likelihood or suffer from label bias problem.

In embodiments, to integrate discriminating power into the action boundary model of the present invention and at the same time keep the generative nature of the action model itself, a DBM is constructed by further augmenting the duration dependency with the contextual information from latent states X and Z so that the switching between actions becomes more discriminative:

where νi, βihave the same interpretation as in Equation (13), and ωijare the additional logistic regression coefficients. When ωijTx=0, no information can be learned from Xtand Zt, and the DBM reduces to a generative one as Equation (13). A similar logistic function has been employed in augmented SLDS, where the main motivation is to distinguish between transitions to different states based on latent variable. Embodiments of the current DBM are specifically designed for locating the boundary between contiguous actions. Such embodiments rely on both real valued and categorical inputs.

In embodiments, as constrained by the STM in Subsection B.2, each action is likely to terminate in stage NQ. Therefore, Di+1may be reset to 1 only when the current action is in this terminating stage, and Equation (15) may be modified as:

In this way, the number of parameters is greatly reduced and the label unbalance problem is also ameliorated. The result is a completion of the construction of the SSM model for continuous action recognition according to embodiments of the present invention, with an overall structure such as the embodiment105depicted inFIG. 1(b).

In embodiments, to learn or train the parameters ν, β and ω, a coordinate descent method may be used to iterate between {ν, β} and ω. For ν and β, given a set of training state sequences {Sn}, the labels for all {Dn} may be easily obtained according to Equation (12) and Equation (13). Then fitting the logistic duration model of Equation (13) equals to performing logistic regression with input feature x=Dtand output y=δ(St+1−St). The action transition probability {aij} may be obtained trivially, by counting the number of transitions from action type i to action type j in the entire training set.

In embodiments, to estimate ωij, let {T(n)}n=1 . . . Nbe the training set, where each data sample T(n)is a realization of all the nodes involved in Equation (15) at a particular time instance t(n)and St(n)=i. Since Xt(n)and Zt(n)are hidden variables, their posterior
p(Zt(n)j|•)=pZ(n)and
p(Xt(n)x|Zt(n)j,•)=(x;μ(n),Σ(n))

are first inferred from single action STM, where the posterior of Xt(n)is approximated by a Gaussian. The estimation of {circumflex over (ω)}ijis obtained by maximizing the expected log likelihood:

The integral in Equation (17) cannot be solved analytically. Instead, in embodiments, an unscented transform is used to approximate the Gaussian(x; μ(n), Σ(n)) using a set of sigma points {xk(n)}k=0 . . . 2M. Therefore, Equation (17) converts to a weighted logistic regression problem with features {xk(n)}, labels {b(n)}, and weights {pZ(n)/(2M+1)}.

In testing, given an observation sequence y1:T, we want to find the MAP action labels Ŝ1:Tand the boundaries defined by {circumflex over (D)}1:T; we are also interested in the study of actions which can be revealed from {circumflex over (Z)}1:T. Evaluating the full posterior p(S1:T, D1:T, Z1:T|y1:T) is a non-trivial job given the complex hierarchy of the model presented herein. In embodiments, a particle filtering may be used for online inference due to its capability in non-linear scenario and temporal scalability. It shall be noted that, in embodiments, the latent variable Xtmay be marginalized by Rao-Blackwellisation, and the computation of particle filtering is significantly reduced since Monte Carlo sampling is conducted in the joint space of (St, Dt, Zt), which has a low dimension and highly compact support.

Formally, in embodiments, the posterior distribution of all the hidden nodes at time t may be decomposed as
p(St,Dt,Zt,Xt|y1:t)=p(St,Dt,Zt|y1:t)p(Xt|St,Dt,Zt,y1:t)  (19)

In Rao-Blackwellised particle filter, a set of NPsamples {(st(n), dt(n), zt(n))}n=1NPand the associated weights {wt(n)}n=1NPare used to approximate the intractable first term in Equation (19), while the second term is represented by {χt(n)(x)}n=1NP, which are analytical distributions conditioned on corresponding samples:

In embodiments of the model presented herein, χt(n)(x)=(x; {circumflex over (x)}t(n), Pt(n)) is a Gaussian distribution. Thus, the posterior may be represented as

where the approximation error approaches to zero as NPincreases to infinite.

Given the samples {(st−1(n), dt−1(n), zt−1(n), χt−1(n)(x))} and weights {wt−1(n)} at time t−1, the posterior of (St, Dt, Zt) at time t may be evaluated as

Equation (23) is essentially the integral of a Gaussian function with a logistic function. Although not solvable analytically, it can be well approximated by a re-parameterized logistic function. In embodiments, it may be approximated according to P. Maragakis, F. Ritort, C. Bustamante, M. Karplus, and G. E. Crooks, “Bayesian estimates of free energies from nonequilibrium work data in the presence of instrument noise,” Journal of Chemical Physics, 129, 2008, which is incorporated in its entirety herein by reference. Nevertheless, it is hard to draw sample from Equation (23). Therefore, in embodiments, new samples (st(n), dt(n), zt(n)) are drawn from a proposal density defined as:
q(St,Dt,Zt|•)=p(St|Dt,st−1(n))p(Zt|St,Dt,zt−1(n))×p(Dt|st−1(n),dt−1(n),zt−1(n),{circumflex over (x)}t−1(n))  (24)

The new sample weights may then be updated as

Once st(n)and zt(n)are obtained, χt(n)(x) is simply updated by Kalman filter. In embodiments, resampling and normalization procedures may be applied after all the samples are updated. In embodiments, resampling and normalization may be performed in like manner as discussed in A. Doucet, N. d. Freitas, K. P. Murphy, and S. J. Russell, “Rao-Blackwellised particle filtering for dynamic Bayesian networks,” in Proceedings of the 16th Conference on Uncertainty in Artificial Intelligence, pages 176-183, 2000, which is incorporated in its entirety herein by reference.

E. Exemplary Training and Inferencing System and Method Embodiments

Given the framework presented above, examples of systems and methods that employ the inventive concepts of the state space model that include logistic duration modeling, substructure transition modeling, and discriminative boundary modeling are presented herein. It shall be noted that these embodiments are presented to elucidate some example applications, and that one skilled in the art shall recognize other applications (both systems and methods), and these additional applications shall be considered within the scope of the current patent document.

FIG. 5depicts a block diagram of a trainer for developing a state space model that includes logistic duration modeling, substructure transition modeling, and discriminative boundary modeling training according to embodiments of the present invention. And,FIG. 6depicts a general methodology for training according to embodiments of the present invention.

As shown inFIG. 5, the model trainer505comprises a frame extractor510, a feature extractor515, and a model trainer520. In embodiments, the frame extractor510received input sensor data545, such as a video although other sensor data may be used, and segments (605) the data into a set of time frames. In embodiments, the feature extractor515receives the segmented data from the frame extractor510and extracts or generates (610) one or more features using at least some of the input sensor data to represent each segmented data frame. In embodiments in which the input sensor data is video data and the time frame data is image frames, an image feature for the image frame is generated to represent the image frame. One skilled in the art shall recognize that numerous ways exist for generating an image feature from an image frame. While any of a number of image features may be employed, in embodiments, the image feature may be an embedded optical flow feature, which is described in commonly assigned and co-pending U.S. patent application Ser. No. 13/405,986, filed on Feb. 27, 2012, entitled “EMBEDDED OPTICAL FLOW FEATURES,” and listing as inventors Jinjun Wang and Jing Xiao, which is incorporated by reference herein in its entirety.

Using the features for the time frames and known action labels from the input training data545, the state, space model trainer520trains the models, including the Logistic Duration Model, the Substructure Transition Model (STM), and the Discriminative Boundary Model (DBM). Thus, in embodiments, the model trainer505may comprise a substructure transition modeler525, a logistic duration modeler530, and a discriminative boundary modeler535. In embodiments, the modelers525-535may utilize one or more of the methods discussed above to train the various models. The resultant output is an embodiment of a probabilistic model550for continuous action recognition that comprises at least two model components: substructure transition model and discriminative boundary model. In embodiments, the substructure transition model component encodes the sparse and global temporal transition prior between action primitives in state-space model to handle the large spatial-temporal variations within an action class. In embodiments, the discriminative boundary model component enforces the action duration constraint in a discriminative way to locate the transition boundaries between actions more accurately. In embodiments, the two model components are integrated into a unified graphical structure to enable effective training and inference. An embodiment of inference is discussed with respect toFIGS. 7 and 8.

FIG. 7depicts a block diagram of a detector or inferencer705that uses a state space model comprising logistic duration modeling, substructure transition modeling, and discriminative boundary modeling according to embodiments of the present invention. And,FIG. 8depicts a general methodology for estimating a sequence of optimal action labels, action boundaries, and action unit indexes given observation data using the detector or inferencer705according to embodiments of the present invention.

As shown inFIG. 7, the inference705comprises a frame extractor510, a feature extractor515, and a trained state space model550. In embodiments, the frame extractor510received input sensor data715, such as a video although other sensor data may be used, and segments (805) the data into a set of time frames. In embodiments, the feature extractor515receives the segmented data from the frame extractor510and extracts or generates (810) one or more features using at least some of the input sensor data to represent each segmented data frame—as previously discussed above. This feature information is supplied to the inference processor710that uses the trained state space model that includes the STM and DBM to output the action labels. In embodiments, the detector.705may also output action boundaries and action unit indexes.

F. Experimental Results

Experimental results on both public and in-house datasets have shown that, with the capability to incorporate additional information that had not been explicitly or efficiently modeled by previous methods, the embodiments presented herein achieved significantly improved performance for continuous action recognition.

An embodiment of the present invention was tested on four datasets for continuous action recognition. In all the experiments, the parameters NQ=3, NZ=5, and NP=200 were used. In embodiments, STM was trained independently for each action using the segmented sequences in training set; then DBM was trained from the inferred terminal stage of each sequence. The overall learning procedure follows expectation-maximization (EM) paradigm where the beginning and terminating stages are initially set as the first and last 15% of each sequence, and the initial action primitives are obtained from K-means clustering. The EM iteration stops when the change in likelihood falls below a threshold. In testing, after the online inference using particle filter, each action boundary may be further adjusted using an off-line inference within a local neighborhood of length40centered at the initial boundary; in this way, the locally “full” posterior in Section D is considered. The recognition performance was evaluated by per-frame accuracy. Contribution from each model component (STM and DBM) was analyzed separately.

1. Public Dataset

The first public dataset used was the IXMAS dataset. The dataset contains 11 actions, each performed 3 times by 10 actors. The videos were acquired using 5 synchronized cameras from different angles, and the actors freely changed their orientation for each acquisition. The dense optical flow in the silhouette area of each subject was calculated, from which Locality-constrained Linear Coding features (LLC) were extracted as the observation in each frame. 32 codewords and 4×4, 2×2, and 1×1 spatial pyramid were used. Table 1 reports the continuous action recognition results, in comparison with Switching LDS (SLDS), Conditional Random Fields (CRF), and Latent-Dynamic CFR (LDCRF). The embodiment of the current invention (and each of its components) achieved recognition accuracy higher than all the other methods by more than 10%.

The second public dataset used was the Carnegie Mellon University Motion Capture Database (CMU MoCap) dataset. For comparison purpose, the results from the complete subset of subject86is reported. The subset has 14 sequences with 122 actions in 8 categories. Quaternion feature was derived from the raw MoCap data as our observation for inference. Table 2 lists the continuous action recognition results, in comparison with the same set of benchmark techniques as in the first experiment, as well as compared with of the approaches in N. Ozay, M. Sznaier, and C. O, “Sequential sparsification for change detection,” in Proc. of Computer Vision and Pattern Recognition (CVPR '08), pp. 1-6, 2008 (hereinafter, “Ref. 1”) and in M. Raptis, K. Wnuk, and S. Soatto, “Spike train driven dynamical models for human actions,” in Proc. of Computer Vision and Pattern Recognition (CVPR '10), pp. 2077-2084, 2010 (hereinafter, “Ref. 2”). Each of the references is incorporated herein by reference in its entirety. Similarly, results from this experiment demonstrated the superior performance of embodiments of the present invention. It shall be noted that, in Table 2, the frame-level accuracy by using DBM alone is a little higher than its combination with STM. This is because there is only one subject in this experiment and no significant variation in substructure is presented in each action type, so temporal duration plays a more important role in recognition. Nevertheless, the result attained by STM+DBM was superior than all benchmark methods.

In addition to the above two public datasets, two in-house datasets were also captured. The actions in these two sets feature stronger hierarchical substructure. The first dataset contains videos of stacking/unstacking three colored boxes, which involves actions of “move-arm”, “pickup,” and “put-down.” Thirteen (13) sequences with 567 actions were recorded in both Red-Green-Blue and depth videos with one Microsoft Kinect sensor. Then object tracking and 3D reconstruction were performed to obtain the 3D trajectories of two hands and three boxes. In this way, an observation sequence in15 was generated. In the experiments, leave-one-out cross-validation was performed on the 13 sequences. The continuous recognition results are listed in Table 3. It shall be noticed that, among the four benchmark techniques, the performance of SLDS and CRF were comparable, while LDCRF achieved the best performance. This is reasonable because during the stacking process, each box can be moved/stacked at any place on the desk, which leads to large spatial variations that cannot be well modeled by a Bayesian Network of only two layers. LDCRF applied a third layer to capture such “latent dynamics,” and hence achieved best accuracy. For embodiments of the present invention, the STM alone brings SLDS to a comparable accuracy to LDCRF because it also models the action substructure. By further incorporating duration information, embodiments of the present invention outperformed all benchmark approaches.

The second in-house dataset was more complicated than the first one. It involved five actions, “move-arm,” “pick-up,” “put-down,” “plug-in,” and “plug-out,” in printer part assembling task. The 3D trajectory of two hands and two printer parts were extracted using the same Kinect sensor system. Eight (8) sequences were recorded and used with leave-one-out cross-validation. As can be seen from Table 4, embodiments of the present invention with both STM and DBM outperformed other benchmark approaches by a large margin.

3. Additional Comparison

To provide more insightful comparison between embodiments of the present invention and other benchmark algorithms,FIG. 9shows two examples of continuous action recognition results from the in-house dataset. The results given by SLDS contain short and frequent switching between incorrect action types. This is caused by the false matching of motion patterns to an incorrect action model. Duration SLDS (dSLDS) and LDCRF eliminate the short transitions by considering additional context information; however, their performances degraded severely around noisy or ambiguous action periods (e.g., the beginning of the sequence inFIG. 9(b)) due to false duration prior or overdependence on discriminative classifier. The STM+DBM approach of the present patent document does not suffer from any of these problems, because STM helps to identify all action classes disregarding their variations, and DBM further helps to improve the precision of boundaries with both generative and discriminative duration knowledge. Another interesting finding shown in the last rows ofFIG. 9(a) andFIG. 9(b) is that the substructure node Ztcan be interpreted by concrete physical meanings. For all the actions in these experiments, we find different object involved in an action corresponds to a different value of Z that has the highest probability in the inferred values {circumflex over (Z)}1:T. Therefore, in addition to estimating action class, we can also associate object with the action by majority voting based on {circumflex over (Z)}1:T. In our experiments, all the inferred object association agree with ground truth.

G. Computing System Implementations

In embodiments, one or more computing system may be configured to perform one or more of the methods, functions, and/or operations presented herein. Systems that implement at least one or more of the methods, functions, and/or operations described herein may comprise an application or applications operating on at least one computing system. The computing system may comprise one or more computers and one or more databases. The computer system may be a single system, a distributed system, a cloud-based computer system, or a combination thereof.

It shall be noted that the present invention may be implemented in any instruction-execution/computing device or system capable of processing data, including, without limitation phones, laptop computers, desktop computers, and servers. The present invention may also be implemented into other computing devices and systems. Furthermore, aspects of the present invention may be implemented in a wide variety of ways including software (including firmware), hardware, or combinations thereof. For example, the functions to practice various aspects of the present invention may be performed by components that are implemented in a wide variety of ways including discrete logic components, one or more application specific integrated circuits (ASICs), and/or program-controlled processors. It shall be noted that the manner in which these items are implemented is not critical to the present invention.

Having described the details of the invention, an exemplary system1000, which may be used to implement one or more aspects of the present invention, will now be described with reference toFIG. 10. As illustrated inFIG. 10, the system includes a central processing unit (CPU)1001that provides computing resources and controls the computer. The CPU1001may be implemented with a microprocessor or the like, and may also include a graphics processor and/or a floating point coprocessor for mathematical computations. The system1000may also include system memory1002, which may be in the form of random-access memory (RAM) and read-only memory (ROM).

A number of controllers and peripheral devices may also be provided, as shown inFIG. 10. An input controller1003represents an interface to various input device(s)1004, such as a keyboard, mouse, or stylus. There may also be a scanner controller1005, which communicates with a scanner1006. The system1000may also include a storage controller1007for interfacing with one or more storage devices1008each of which includes a storage medium such as magnetic tape or disk, or an optical medium that might be used to record programs of instructions for operating systems, utilities and applications which may include embodiments of programs that implement various aspects of the present invention. Storage device(s)1008may also be used to store processed data or data to be processed in accordance with the invention. The system1000may also include a display controller1009for providing an interface to a display device1011, which may be a cathode ray tube (CRT), a thin film transistor (TFT) display, or other type of display. The system1000may also include a printer controller1012for communicating with a printer1013. A communications controller1014may interface with one or more communication devices1015, which enables the system1000to connect to remote devices through any of a variety of networks including the Internet, a local area network (LAN), a wide area network (WAN), or through any suitable electromagnetic carrier signals including infrared signals.