Patent Description:
<NPL>" describes the problem of human action recognition using 3D reconstruction data. 3D reconstruction techniques are employed for addressing two of the most challenging issues related to human action recognition in the general case, namely view variance (i.e., when the same action is observed from different viewpoints) and the presence of (self-) occlusions (i.e., when for a given point of view a body part of an individual conceals another body part of the same or another subject). The main contributions of the paper are summarized as follows. The first one is a detailed examination of the use of 3D reconstruction data for performing human action recognition. The latter includes the introduction of appropriate local/global flow/shape descriptors, extensive experiments in challenging publicly available datasets and exhaustive comparisons with state-of-art approaches. The second one is a new local-level 3D flow descriptor, which incorporates spatial and surface information in the flow representation and efficiently handles the problem of defining 3D orientation at every local neighborhood. The third one is a new global-level 3D flow descriptor that efficiently encodes the global motion characteristics in a compact way. The fourth one is a novel global temporal-shape descriptor that extends the notion of 3D shape descriptions for action recognition, by incorporating the temporal dimension. The proposed descriptor efficiently addresses the inherent problems of temporal alignment and compact representation, while also being robust in the presence of noise (compared with similar tracking-based methods of the literature). Overall, the paper significantly improves the state-of-art performance and introduces new research directions in the field of 3D action recognition, following the recent development and wide-spread use of portable, affordable, high-quality and accurate motion capturing devices (e.g., Microsoft Kinect).

<CIT> describes a method for performing health risk assessments for a patient in a home or medical facility. In various embodiments, the method comprises compiling depth image data from at least one depth camera associated with a particular patient, and generating at least one three-dimensional object based on the depth image data. The method additionally includes identifying a walking sequence from the at least one three-dimensional object, analyzing the walking sequence to generate one or more parameters, and performing at least one health risk assessment based on the one or more parameters to determine a health risk assessment score. The method further comprises sending an alert message to at least one caregiver when the at least one health risk assessment indicates the occurrence of an incident (i.e., the recorded depth image data) denoting that the patient has fallen or that there exists a high risk of the patient falling.

<NPL>" describes how accidental falls have been identified as a cause of mortality for elders who live alone around the globe. Following a fall, additional injury can be sustained if proper fall recovery techniques are not followed. These secondary complications can be reduced if the person had access to safe recovery procedures or were assisted, either by a person or a robot. It proposes a framework for in situ robotic assistance for post fall recovery scenarios. In order to assist autonomously robots need to recognize an individual's posture and subactivities (e.g., falling, rolling, move to hands and knees, crawling, and push up through legs, sitting or standing). Human body skeleton tracking through RGB-D pose estimation methods fail to identify the body parts during key phases of fall recovery due to high occlusion rates in fallen, and recovering, postures. To address this issue, it investigated how low-level image features can be leveraged to recognize an individual's subactivities. Depth cuboid similarity features (DCSFs) approach was improved with M-partitioned histograms of depth cuboid prototypes, integration of activity progression direction, and outlier spatiotemporal interest point removal. The modified DCSF algorithm was evaluated on a unique RGB-D multiview dataset, achieving <NUM> ± <NUM>% accuracy in the extensive <NUM> (C15 <NUM>) combinations of trainingtest groups of <NUM> subjects in <NUM> trials. This result was significantly larger than the nearest competitor, and faster in the training phase. This work could lead to more accurate in situ robotic assistance for fall recovery, saving lives for victims of falls.

<NPL>" describes how human action recognition system is fundamental of human activity and behavior recognition, especially for video analysis technologies. The paper introduces an improvement method for human action recognition proposed by P. Chawalitsittikul et al. The actions from RGBD multi-views, taken from cameras at different static-viewpoints in the overlapping Area of Interest, are fused at high-level decision. The empirical fusion model is derived from performances of action recognition in various viewpoints: front, slant, side, back-slant, and back. The results shown that the fusion model improves significantly the accuracy using only one more camera.

<NPL>" describes a new pipeline for live multi-view performance capture, generating temporally coherent high-quality reconstructions in real-time. The algorithm supports both incremental reconstruction, improving the surface estimation over time, as well as parameterizing the nonrigid scene motion. The approach is highly robust to both large frame-to-frame motion and topology changes, allowing us to reconstruct extremely challenging scenes. It demonstrates advantages over related real-time techniques that either deform an online generated template or continually fuse depth data nonrigidly into a single reference model. Finally, it shows geometric reconstruction results on par with offline methods which require orders of magnitude more processing time and many more RGBD cameras.

Aspects of the present invention are provided in the accompanying claims.

The accompanying drawings illustrate implementations of the concepts conveyed in the present patent. Features of the illustrated implementations can be more readily understood by reference to the following description taken in conjunction with the accompanying drawings. Like reference numbers in the various drawings are used wherever feasible to indicate like elements. In some cases, parentheticals are utilized after a reference number to distinguish like elements. Use of the reference number without the associated parenthetical is generic to the element. Further, the left-most numeral of each reference number conveys the figure and associated discussion where the reference number is first introduced.

Recognizing the actions of people in a crowded and cluttered environment is a challenging computer vision task. This description relates to tracking actions of people and/or objects utilizing multiple three-dimensional (3D) cameras. Depth data from the multiple 3D cameras are used to determine which voxels in an environment are occupied by a person or object. Voxel occupancy are used to construct solid volume data, as opposed to simply outlining surfaces of people or objects. Taken one step further, collecting the depth data from the multiple 3D cameras over time are used to perform 4D dynamic solid modeling of the whole space. With the added dimension of time, 4D dynamic solid modeling can efficiently and accurately identify real-time actions and/or behaviors of people, pets, robots, cars, etc., and their interactions with objects in the environment. The present 4D dynamic solid modeling concepts can be implemented in almost any use case scenario, even including large-scale, cluttered environments, such as a crowded store, a busy factory, or even a fast-paced city block. For purposes of explanation, the description first turns to a relatively simple office scenario.

<FIG>, <FIG>, <FIG>, <FIG>, and <FIG> collectively show an example 4D dynamic solid modeling (4D DSM) scenario <NUM>. In this case, <FIG> shows an environment <NUM> that includes three people <NUM> (e.g., persons). In this simple office scene, the environment <NUM> also contains various objects <NUM>, such as a cup, a computer, a keyboard, papers, etc. In this example, some objects are designated with specificity to aid in the following discussion, including a table <NUM>, a chair <NUM>, a couch <NUM>, a coffee table <NUM>, and a desk <NUM>. The scene shown in <FIG> can be thought of as a particular time point (e.g., snapshot in time), Instance One. People <NUM> and/or objects <NUM> are also referred to as subjects <NUM>.

In 4D dynamic solid modeling scenario <NUM>, the environment <NUM> can also include various cameras <NUM>. In <FIG>, four cameras <NUM> are shown. Camera <NUM>(<NUM>) is mounted overhead in the foreground of the scene, and aimed into the drawing page. Cameras <NUM>(<NUM>) and <NUM>(<NUM>) are mounted overhead on the right and left side relative to the drawing page, respectively, and aimed toward the office scene. Camera <NUM>(<NUM>) is mounted in the background of the scene. As such, the direction of the view shown in <FIG> is roughly similar to that of camera <NUM>(<NUM>). Cameras <NUM> are 3D cameras, which can employ any type of 3D technology, such as stereo cameras, structured light, time of flight, etc., as well as capture color images. In this case, the 3D cameras can collectively capture depth data of the scene over time (e.g., 4D data). The depth data collected over time is used to determine successive 3D solid volume descriptions of the scene. Stated another way, the cameras <NUM> can be used to perform 4D dynamic solid volume sensing of the environment <NUM>. The 4D dynamic solid volume data is analyzed to identify actions of people and/or objects in the environment <NUM>.

<FIG> show environment <NUM> from the different viewpoints of cameras <NUM>. In this case, <FIG> represents the view from camera <NUM>(<NUM>), <FIG> represents the view from camera <NUM>(<NUM>), <FIG> represents the view from camera <NUM>(<NUM>), and <FIG> represents the view from camera <NUM>(<NUM>). The images shown in <FIG> also represent Instance One, having all been captured at approximately the same time. Note that the multiple camera angles provide relatively extensive coverage of the people <NUM> and objects <NUM> in the environment <NUM>. For example, in <FIG> the view of person <NUM>(<NUM>) is somewhat obscured, since person <NUM>(<NUM>) is partially between the camera <NUM>(<NUM>) and person <NUM>(<NUM>). However, the data captured by cameras <NUM>(<NUM>), <NUM>(<NUM>), and <NUM>(<NUM>) can collectively overcome, or at least partially remedy, deficiencies in the view of camera <NUM>(<NUM>) in Instance One.

By combining the depth data collected from the different camera viewpoints depicted in <FIG>, 4D dynamic solid modeling concepts (e.g., algorithms) are used to determine which voxels in environment <NUM> are occupied by a person or other object. Once voxel occupancy is determined, a 3D solid volume representation <NUM> of environment <NUM> can be created. <FIG> collectively show a 3D solid volume representation <NUM> of environment <NUM>. As such, the 3D solid volume representation <NUM> shown in <FIG> is thought of as a 3D map of voxel occupancy at the Instance One time point.

In some implementations, 4D dynamic solid modeling can be used to track people and/or objects. In order to track a person in 4D dynamic solid modeling scenario <NUM>, the environment <NUM> is partitioned into partial volumes <NUM>, shown in <FIG>. In this example, the partial volumes <NUM> have the same height, as if they all extend from the floor to the ceiling of environment <NUM>. In this case, each partial volume <NUM> contains a portion of the 3D solid volume representation <NUM> that includes one of the people <NUM>. For instance, partial volume <NUM>(<NUM>) is roughly placed around the portion of the 3D solid volume representation <NUM> that includes person <NUM>(<NUM>). The portions of the 3D solid volume representation <NUM> included in the partial volumes <NUM> can also correspond to other objects or portions of objects. For instance, partial volume <NUM>(<NUM>) also includes at least a portion of desk <NUM>, which is proximate to person <NUM>(<NUM>) in Instance One. Partial volumes <NUM> can overlap, thereby containing a common portion of the 3D solid volume representation <NUM>. In this example, partial volume <NUM>(<NUM>) and partial volume <NUM>(<NUM>) overlap, as indicated at <NUM>.

<FIG> show the partial volumes <NUM> separated from each other. In this example, where the partial volumes <NUM> overlap, the common portion of the 3D solid volume representation <NUM> is included in both of the separated partial volumes <NUM>. For example, separated partial volume <NUM>(<NUM>) and partial volume <NUM>(<NUM>) both contain a common portion of the 3D solid volume representation <NUM> of the coffee table <NUM>, indicated at <NUM>.

In some cases, viewing the partial volumes <NUM> separately can help simplify the problem of tracking and/or recognizing people's actions or behaviors. For instance, viewing the partial volumes <NUM> separately can focus on the movements and/or actions of a single person. Also, processing the partial volumes <NUM> separately from the whole 3D solid volume representation <NUM> can reduce an amount of processing resources needed to solve the problems of tracking the person and determining the action(s) of the person.

To bring the 4D dynamic solid modeling scenario <NUM> from 3D to 4D, images from an additional time point can be added. <FIG> depicts the same environment <NUM>, but now at Instance Two, a different time point from Instance One. In Instance Two, the people <NUM> are located at different positions within the scene. For example, in Instance One person <NUM>(<NUM>) is sitting in chair <NUM> at the desk <NUM>, holding some papers. In Instance Two, person <NUM>(<NUM>) is standing behind chair <NUM>, and person <NUM>(<NUM>) is holding the papers. At Instance Two, the multiple cameras <NUM> can again be used to capture depth data from multiple views of environment <NUM>, and a new 3D solid volume representation of environment <NUM> can be constructed. For sake of brevity, the detail of the new 3D solid volume representation will not be repeated for Instance Two. To summarize, by repeatedly capturing depth data from the multiple cameras <NUM> at different time points, and constructing 3D solid volume representations for each time point captured, 4D dynamic solid modeling is used to track people and/or objects over time, as described further relative to <FIG>.

<FIG> illustrate environment <NUM> from an overhead (e.g., top down) view. As such, <FIG> show the three people <NUM> and objects <NUM>, including for instance the table <NUM>, couch <NUM>, and desk <NUM> introduced above. <FIG> each illustrate environment <NUM> at a different time point. For example, <FIG> shows Instance One, <FIG> shows Instance Two, and <FIG> shows a third time point, Instance Three. <FIG> also includes the partial volumes <NUM> that were introduced above relative to <FIG>. Note that since <FIG> represent different time points, the people <NUM> and their respective partial volumes <NUM> have different locations in each FIG. Therefore, <FIG> are an illustration of how the people <NUM> can be tracked over time using 4D dynamic solid modeling concepts. With 4D dynamic solid modeling, any partial volume can be followed over time, whether stationary or moving, to continually identify actions of people and/or objects within the partial volume.

<FIG> provides a visualization of application of 4D dynamic solid modeling concepts. <FIG> includes the partial volumes <NUM> separated from each other, as described above relative to <FIG>. <FIG> also includes action recognition technique <NUM> and recognized actions <NUM>. As shown in <FIG>, partial volume <NUM>(<NUM>) is subjected to action recognition technique <NUM>(<NUM>). The result of action recognition technique <NUM>(<NUM>) is recognized action <NUM>(<NUM>) of person <NUM>(<NUM>). Similarly, the result of action recognition technique <NUM>(<NUM>) is recognized action <NUM>(<NUM>), and the result of action recognition technique <NUM>(<NUM>) is recognized action <NUM>(<NUM>) of person <NUM>(<NUM>). In some cases, action recognition technique <NUM> can be a same technique applied repeatedly to all of the partial volumes <NUM>. In other cases the applied action recognition technique <NUM> can vary between partial volumes <NUM> and/or vary over time for a partial volume <NUM>. Further detail of an example action recognition technique <NUM> will be described relative to <FIG>, below.

<FIG> shows additional examples of 3D solid volume representations <NUM> in partial volumes <NUM> with people performing various actions. The examples in <FIG> include 3D solid volume representations <NUM> of people bending, drinking, lifting, pushing/pulling, opening a drawer, reading, waving, and clapping, respectively. In these examples, the partial volumes <NUM> are roughly centered on a person, and many of the partial volumes <NUM> include portions of other objects and/or background material. Not only can 4D dynamic solid modeling concepts reliably recognize actions of people despite such clutter in an environment, in some cases these other objects and/or materials can help with recognition of the actions. For instance, detection of a chair can be used as contextual information to help recognize that a person is performing the action of sitting. The examples in <FIG> are just a few examples of the myriad of positions and/or poses related to actions that is recognized with 4D dynamic solid modeling concepts.

Referring again to <FIG>, with the tracking of the partial volumes <NUM> over time, the action recognition technique <NUM> introduced in <FIG> is repeatedly applied over time to continue to update the recognized actions <NUM> of the people <NUM> in their respective partial volumes <NUM>. For example, as time passes from the Instance One (<FIG>) to Instance Two (<FIG>), action recognition technique <NUM>(<NUM>) can update the recognized action <NUM>(<NUM>) of person <NUM>(<NUM>) from sitting to standing. Action recognition techniques <NUM>(<NUM>) and <NUM>(<NUM>) can similarly continue to update recognized actions <NUM>(<NUM>) and <NUM>(<NUM>). In this manner, 4D dynamic solid modeling concepts can be used to build an understanding of a person's actions and/or behaviors over time.

Taken one step further, the person's actions and/or behaviors within his/her respective partial volume can be placed back into the context of the broader environment <NUM>. Using the combined recognized actions <NUM> of the people <NUM> in the environment <NUM>, an understanding of the interactions of people and/or objects can be built. For instance, 4D dynamic solid modeling can determine that in Instance One (e.g., <FIG>), person <NUM>(<NUM>) is interacting with person <NUM>(<NUM>) by handing person <NUM>(<NUM>) a cup. 4D dynamic solid modeling can also determine that person <NUM>(<NUM>), sitting at the desk <NUM>, does not appear to be interacting with person <NUM>(<NUM>) or person <NUM>(<NUM>). In this manner, 4D dynamic solid modeling can begin to mimic human vision, taking in a holistic view of an environment to figure out what is happening in a scene.

The 4D dynamic solid modeling scenario <NUM> described above illustrates how a 4D scan, using depth data from multiple cameras over time, is used to produce a 4D solid volume model of the scene. Rather than simply outlining the surfaces of people or objects, the 4D solid volume model describes the voxel occupancy for the space. Stated another way, the 4D solid volume model can describe the internal fillings of the entire space. With a 4D solid volume model, potentially every detail about people and their environment can be captured, even in a large-scale, cluttered scene. This rich 4D detail is used to reliably track the actions of people and/or objects in real-world settings and build an understanding of their behaviors and interactions.

The uses for such a robust understanding of people's actions and behaviors are practically limitless. Workplaces could improve efficiency and/or comfort by optimizing the movement patterns of workers. Stores could improve product placement by better understanding the interaction of shoppers with products. Factories could improve safety by limiting the proximity of humans to potentially dangerous movements of large, industrial robots. Traffic accidents could be avoided by monitoring the flow of cars and/or pedestrians on a city street. Even our own homes could be equipped to respond to our activities by anticipating needs for lighting, temperature, music, letting a pet in or out, adding items to a grocery list, etc. As such, 4D dynamic solid modeling concepts can be an integral part of smart homes, smart stores, smart factories - a smarter world.

<FIG> illustrate example techniques (e.g., methods) for performing 4D dynamic solid modeling concepts. <FIG> can include example technique <NUM>. In <FIG>, blocks <NUM>-<NUM> outline the steps of technique <NUM>, which are described in detail below. <FIG> provides additional detail regarding block <NUM>. <FIG> provides additional detail regarding block <NUM>. <FIG> provides additional detail regarding block <NUM>.

In some implementations, aspects of the example techniques described relative to <FIG> can be similar to aspects of the 4D dynamic solid modeling scenario <NUM> described above relative to <FIG>. As such, occasional reference will be made to previous FIGS. to assist the understanding of the reader.

As shown in <FIG>, at block <NUM>, technique <NUM> receives depth data of an environment sensed by multiple 3D cameras over time. The cameras are similar to cameras <NUM> shown in the example in <FIG>. In one implementation, the cameras can be Kinect™ brand cameras from Microsoft Corporation.

At block <NUM>, technique <NUM> determines voxel occupancy of the environment from the depth data. At block <NUM>, technique <NUM> constructs a 3D solid volume representation using the voxel occupancy. In some implementations, a 3D solid volume construction algorithm can perform blocks <NUM> and/or <NUM> of technique <NUM>.

Referring to block <NUM>, determining voxel occupancy can include partitioning the environment into voxels. Potentially any environment is partitioned into voxels. Furthermore, the environment is mapped using a world coordinate system, such as the x (horizontal), y (horizontal), and/or z (vertical) coordinates shown in the example in <FIG>, for example. The cameras and/or depth data can be calibrated with respect to the world coordinate system. Once the environment is mapped, each voxel can be located by its respective coordinates. Voxel occupancy is determined based on whether each of the located voxels contains part of a subject (e.g., person and/or object).

In some cases, the environment is partitioned into voxels with a cube shape. For instance, a voxel can have a cube shape measuring <NUM> millimeters per side. In another instance, a voxel can measure <NUM> centimeters per side. Other sizes, shapes, methods for partitioning an environment, and/or methods for mapping voxels in an environment are contemplated.

In one implementation, given a set of calibrated RGBD images, voxel center coordinates can be denoted as (xi, yi, zi), where i = <NUM>. A number of cameras can be M. Extrinsic matrices of the cameras can be [Rj|tj], where j = <NUM>. M, Rj is a rotation matrix, tj is a translation vector, and the intrinsic matrices are Kj. The depth images from the cameras can be denoted as D<NUM>,. In the following, <NUM> and <NUM> can represent false (e.g., an unoccupied voxel) and true (e.g., an occupied voxel), respectively. The occupancy of the voxel at (xj, yi, zi) from camera j can be computed as: <MAT> where <MAT> and <MAT> are the third row of Rj and tj. Oj(i) can also be conditioned on the camera field of view. For example, if the projection Kj [Rj | tj ][xi, yi, zi, <NUM>]T is outside of the field of view, Oj(i) can be set to <NUM>. Thus, the occupancy O(i) of the voxel i can be the intersection of Oj(i) from all the M cameras: <MAT>.

Referring to block <NUM>, a 3D solid volume representation of the environment is constructed using the volume occupancy. In some cases, only the volume seen from a particular camera is carved out. This aspect can allow construction of a 3D solid volume representation even where the fields of view of different depth cameras do not have overlap.

In some implementations, the following two techniques can further improve quality of 3D solid volume representation: <NUM>) an orthographic top-down view of the point cloud in the volume can be used as a mask to remove small "tails" introduced at camera boundary regions; and <NUM>) small misses in synchronization among cameras can be mitigated. In some cases, poor synchronization among cameras can lead to vanishing of thin structures in 4D volumes. A best-effort fashion can include extracting frames from all the cameras linked together into a local network. For example, with fast moving body parts (e.g., arms), small misses in synchronization may occur. To remedy this issue, all the points from the depth cameras can be injected into the solid volume. For example, O(i) can be set to one where there is a point in the voxel i. These voxels can be on the scene surface and the other voxels can be internal voxels. In this example, the holistic property of the 4D volume can produce reliable action recognition.

In some cases, directly computing 4D solid volumes using a CPU can be resource prohibitive due to the large number of voxels. For example, the example environment <NUM> described relative to <FIG> above can have a volume of <NUM> x <NUM> x <NUM> voxels. Alternatively, pre-computation can be performed for <MAT> and Kj [Rj | tj ][xi, yi, zi, <NUM>]T. Then, comparison operations can be performed in parallel in a GPU. Similarly, point cloud filling and top-down carving can also be performed in parallel. In this example, real-time 4D solid volume modeling of environment <NUM> can be performed with as little as <NUM> to <NUM> percent of GPU usage.

At block <NUM>, technique <NUM> selects a subject in the 3D solid volume representation. As described above, the subject is a person and/or object to be tracked in a scene. In some cases, 4D dynamic solid modeling can include scanning cluttered, crowded, and/or fast-moving scenes. In conditions such as these, direct action recognition can be difficult, with potential for inaccurate results and/or intense computing resource requirements. Computing resources can be conserved and results can be improved by focusing attention on selected subjects.

To select a subject using the 3D solid volume representation, as a first step, subject candidates are detected. For example, a subject candidate detection algorithm can be employed. Although not claimed in the appended claims, a sweeping volume solution could be used to detect subject candidates; this approach can have high complexity. Alternatively, a light-weight solution, which is not claimed in the appended claims, can be used. A top-down envelope image can be processed to detect subject candidates. For example, f(m,n,k) can be the volume data. In this example, m,n can be x,y coordinates, and k can be the z coordinate. Here, z = <NUM> can be the ground plane. The top-down envelope can be g(m,n) = maxk(ϕ(f(m,n,k))), where ϕ(f(m,n,k)) = k if f(m,n,k) > <NUM> and otherwise ϕ(f(m,n,k))=<NUM>. In some cases, each potential subject can correspond to at least one local maximum on g. A simple Gaussian filter is used to extract the subject candidates. A subject candidate is detected by locating a local maximum with a given width and height. The local maxima can be found on the Gaussian-filtered, top-down envelope using non-maximum suppression. Additional voxel attributes, such as color and/or multi-resolution volume data, can be used to assist in detecting subject candidates.

Once subject candidates are detected, a partial volume (similar to the partial volumes <NUM> introduced above relative to <FIG>) is established around each subject candidate. For example, the subject candidate detection algorithm can also establish a corresponding partial volume for each subject candidate. In some cases the partial volumes can be cuboids. In some cases a height of the partial volumes can be the overall height of the mapped environment. In this example depicted in <FIG>, a partial volume can have a volume of <NUM>×<NUM>×<NUM> voxels, where the voxels measure <NUM> centimeters per side. This example volume can be large enough to cover a person with different poses. Other sizes and/or shapes of partial volumes are contemplated.

A subject classifier algorithm is used to classify whether each partial volume contains a subject. Machine learning with a trained model is used for classification. For example, a 3D subject classifier convolutional neural network (CNN) is used for classification. For instance, a 3D people classifier CNN trained on labeled training data with people can be used to classify whether a partial volume contains a person. In other cases, other models could be used to classify whether partial volumes contain other subjects of interest, such as pets, robots, cars, etc..

The structure of an example 3D subject classifier CNN <NUM> is shown in <FIG>. The 3D subject classifier CNN <NUM> can contain a sequence of 3D convolution layers <NUM>, rectified linear units (ReLUs), and pooling layers <NUM> (e.g., max-pooling layers) to extract features from the partial volume. In <FIG>, to avoid clutter on the drawing page, only one instance of a 3D convolution layer <NUM> and a pooling layer <NUM> are designated, and the ReLUs are not shown. The features are then fed into a fully connected network for people classification. The 3D subject classifier CNN <NUM> can give a probability of each partial volume containing a subject. In some cases, tracking partial volumes over time can remove false subject detections and/or smooth out missing subject detections, therefore improving the detection accuracy. Even with just a few thousand frames of training data, in some cases the 3D subject classifier CNN <NUM> can achieve relatively high people classification accuracy for the purposes of 4D dynamic solid modeling. The results from 3D subject classifier CNN <NUM> can be used to select subjects to be tracked in the next step of technique <NUM>.

Referring again to <FIG>, at block <NUM>, technique <NUM> tracks the selected subjects using the depth data sensed over time with a subject tracking algorithm. In some cases, a subject tracking algorithm can be represented by the example tracking graph <NUM> shown in <FIG>. In this example, subject tracking can be formulated as a path following problem. For instance, subject tracking can include construction of a trajectory for a selected subject in a current frame t and the next n frames over time. In one example, n can be a small number, e.g., three. A small number of frames can introduce a short delay, which can improve reliability of subject tracking.

As shown in <FIG>, tracking graph <NUM> can include three kinds of nodes: trajectory nodes <NUM> (rectangle shape) represent trajectories already formed, prediction nodes <NUM> (pentagon shape), candidate nodes <NUM> (circle shape), and edges <NUM>. Only one of each type of element is labeled in <FIG> to avoid clutter on the drawing page. The number of prediction nodes <NUM> can equal the number of candidate nodes <NUM> plus the number of prediction nodes <NUM> at a previous time point. The edges <NUM> in the graph can indicate possible matches between nodes. Edge weights can be determined by a difference in probabilities between the 3D subject classifier CNN <NUM>, a Euclidean distance, a voxel occupancy volume difference, and/or a color histogram difference between neighboring nodes. The trajectory nodes <NUM> can also have a weight inversely proportional to a trajectory length. In some cases, the subject tracking algorithm can include finding an extension of each trajectory from time t-<NUM> to t+n, so that the paths pass each trajectory node <NUM> and all the paths are node disjoint.

In some cases, the subject tracking algorithm can be reduced to a min-cost flow problem and solved using a polynomial algorithm. For instance, when tracking a person, each trajectory can be extended to the neighboring nodes within a radius dL, which can be determined by the max-speed of a person and the frame rate of the subject tracking algorithm. A gating constraint can also speed up the (potentially optimal) path search.

In this example subject tracking algorithm, after the path search, each existing trajectory can be extended by one-unit length. In a person tracking instance, the trajectories with a low people score can be removed. Here, the people score can be defined as the weighted sum of a current people probability and a previous people score. Also, new trajectories can be included for each candidate node at time t that is not on any path. The new trajectories can be used to form a new graph for the next time instant. The procedure can be repeated for each new video frame received.

In some cases, <FIG> can be viewed as an example result of the subject tracking algorithm. <FIG> can represent 4D tracking over a few thousand image frames, for example. The subject tracking algorithm can be robust against clutter and crowd. For instance, the subject tracking algorithm can perform in relatively complex interaction cases, such as two people hugging, without tracking loss.

Referring again to <FIG>, at block <NUM>, technique <NUM> recognizes an action of the tracked subject using the 3D solid volume representation. For example, an action recognition procedure can include analyzing how the 3D solid volume representation of a person (e.g., a person volume) evolves over time to determine a recognized action of the person (see for example <FIG>, above). Since the present 4D dynamic solid modeling concepts include determining voxel occupancy for potentially the entire environment, the 3D solid volume representation within any given partial volume can include rich contextual information. This contextual information can be valuable background context for recognizing an action. For instance, a chair underneath a person can be used to infer that the person is sitting. In some cases, body poses, movement of body parts, and/or objects a person is handling can be viewed as clues to infer the action of a person. In other implementations, the position and/or speed of a subject could also be used in action recognition.

An action recognition algorithm includes use of a trained model to recognize actions. The trained model is a deep convolutional neural network. For example, <FIG> illustrates an example model architecture, termed Action4D-Net <NUM>. Inputs to Action4D-Net <NUM> can be a sequence of partial volumes automatically extracted from the subject selection and subject tracking procedures described above relative to blocks <NUM> and <NUM> of <FIG>.

As shown in <FIG>, in Action4D-Net <NUM>, inputs can go through a 3D CNN sequence <NUM>. The 3D CNN sequence <NUM> can include several 3D convolution layers followed by 3D max-pooling layers which can produce action features. In some implementations, an auxiliary attention module <NUM> can be used to generate local features. Also, a global module <NUM> with global max-pooling can be used to generate global features. Both the local and the global features for each time instant can be concatenated. In this example, the concatenated features can be inputs into a classifier, such as a Recurrent Neural Network (RNN) <NUM>. RNN <NUM> can be a Long Short-Term Memory (LSTM) cell, for instance. The RNN <NUM> can be used to aggregate temporal information for a final action classification.

The auxiliary attention module <NUM> can improve performance of action recognition by mimicking the ability of humans to focus attention on different regions when recognizing different actions. For instance, when recognizing the action of book reading, humans generally focus on hands of a subject and the book in his/her hands. While recognizing the action of drinking water, humans shift focus to the mouth area of the subject. The auxiliary attention module <NUM> is used to mimic this attention focus. In particular, the auxiliary attention module <NUM> is able to automatically discover relevance between different inputs at a given context. The auxiliary attention module <NUM> can be employed to automatically learn the (potentially) most relevant local sub-volumes for a given action.

For example, <MAT> can be the output from the last 3D convolution layer, where F is the number of filters and L, W, and H are the size of the 3D output. In particular, each location in the 3D output can be represented as <MAT> for <NUM> ≤ i ≤ L, <NUM> ≤ j ≤ W and <NUM> ≤ k ≤ H. The attention weights for all vijk can be computed as: <MAT> <MAT>
where <MAT> can be the attention weights, <MAT> can be the weight matrix to be learned, and <MAT> can be the previous hidden state of size D from the RNN. Here, the network can automatically discover relevance of different sub-volumes for different actions. Next, the local feature v can be computed as a weighted sum of all the sub-volume features vijk.

At global module <NUM>, global max-pooling can be employed to extract global features as extra information for action recognition. For instance, 3D solid volume representations of people sitting vs. kicking can look quite different. These different actions can be captured by the global features of the partial volumes. A 3D convolution layer can be used, followed by a global pooling layer, to obtain the global feature g. Subsequently, the global feature g and the local attention feature v can be supplied to the LSTM cell to capture temporal dependencies. An action classification model, which can be a Multi-Layer Perceptron (MLP), for example, can take a hidden state from the LSTM cell as input to generate recognized actions.

Referring again to <FIG>, at block <NUM>, technique <NUM> outputs the recognized action. The recognized action can be used in a variety of scenarios, some of which have been suggested earlier. The recognized actions can be combined to consider interactions of people and/or objects. The recognized actions can be analyzed to understand behaviors of people and/or objects over time.

The described methods can be performed by the systems and/or elements described above and/or below, and/or by other 4D dynamic solid modeling devices and/or systems.

The order in which the methods are described is not intended to be construed as a limitation, and any number of the described acts can be combined in any order to implement the method, or an alternate method. Furthermore, the method can be implemented in any suitable hardware, software, firmware, or combination thereof, such that a device can implement the method. In one case, the method is stored on one or more computer-readable storage medium/media as a set of instructions (e.g., computer-readable instructions or computer-executable instructions) such that execution by a processor of a computing device causes the computing device to perform the method.

Table <NUM>, provided below, shows results of the present 4D dynamic solid modeling concepts compared to the existing computer vision methods. In order to introduce the results shown in Table <NUM>, following are brief descriptions of the technical problem of action recognition by computer vision, and the existing computer vision approaches to solving this technical problem.

In general, human vision is good at recognizing subtle actions. Computer vision can have difficulty recognizing and categorizing actions with the robustness and accuracy of human vision. The difficulty can be caused by the variations of the visual inputs, such as a crowded and cluttered environment. For example, in a video of an environment, people may have different clothing, different body shapes, and/or may perform the same action in slightly different ways. The environment captured in the video may be crowded with other people or objects that create occlusions, in other words partially blocking a view of a person performing an action. A crowded or cluttered environment can also make it difficult to segment out (e.g., distinguish) a person from other people or objects. In another example, a viewing angle of a video camera may be different from a viewing angle of a training video with which a computer vision method is trained. In this example, due to the different viewing angle, an action in the video may look quite different from the same action shown in the training video, and computer vision may have trouble recognizing the action.

With existing computer vision approaches, the data collection and/or processing requirements to produce reliable results can be onerous. For example, successful action recognition can require deep learning methods based on multiple data streams, such as color, motion, body part heat maps, and/or finding actions in spatial-temporal 3D volumes. In other cases, in order to produce reliable results, training data are required to include a wide variation of camera settings, people's clothing, object appearances, and backgrounds. Other approaches require special hardware for high quality stereo imaging, and/or complex equipment calibration methods for precise point alignment. Still other approaches require special blue/green or static backgrounds to be able to single out people and determine their actions, making these approaches impractical in the real world. Some approaches include semantic segmentation to differentiate a person from a background before trying to recognize an action of the person. However, semantic segmentation may lose body parts or include other background objects. For this reason, semantic segmentation errors can cause action recognition failures. Many methods are camera view dependent - if the view of the camera is different from the trained model, the model must be retrained before actions can be recognized.

The results shown in Table <NUM> were produced with an experimental setup intended to approximate a real-world environment, without relying on the impractical requirements described above. The experimental setup included a cluttered scene with multiple people. The scene included various objects such as a sofa, tables, chairs, boxes, drawers, cups, and books. The people had different body shapes, gender, and heights. The dataset included the people performing <NUM> actions, such as drinking, clapping, reading a book, calling, playing with a phone, bending, squatting, waving hands, sitting, pointing, lifting, opening a drawer, pull/pushing, eating, yawning, and kicking. As indicated in Table <NUM>, the experiment was performed with people in five groupings.

In the experimental setup, four RGBD cameras were used to capture videos of the environment. Some of the videos were used to train models, and some of the videos were used for testing. Each tracked person in each video frame was assigned an action label. During the experiment, an action classification was determined to be correct where the predicted action label for a particular person in each video frame matched the assigned action label. The match was accepted within a window of plus/minus three successive video frames relative to the particular video frame.

The existing computer vision approaches tested included ShapeContext, Moments, Color + Depth, Skeleton, and PointNet. Each of these existing computer vision approaches are briefly introduced below.

ShapeContext, or 3D Shape context, is a 3D version of a shape context descriptor, where the context of a shape is used to model whole body configuration in an effort to better recognize actions of the body. In general, hand-crafted features such as shape context can be less robust than learned features from deep learning, especially when there is strong background clutter. In this experiment, ShapeContext had the height axis and the angle axis uniformly partitioned, and the radial axis logarithmically partitioned. ShapeContext can have different number of bins (e.g., <NUM> bins). For ShapeContext, a deep network was used in which the input was the 3D shape context descriptors. The deep network used a Long Short-Term Memory (LSTM) network to aggregate temporal information.

Moment is another example of a shape descriptor, and another example of a hand-crafted feature that can be less robust than learned features, especially when there is strong background clutter. In this experiment, raw moments up to order <NUM> were used. Each element of a moment vector was computed as Σx,y,z(x - xc)p(y - yc)q(z - zc)r, where (x,y,z) were the coordinates of the occupied voxels and (xc, yc, zc) was the volume center. Similar to the above ShapeContext approach, the moment descriptor was fed into a CNN for action recognition.

Relating to Skeleton, images are analyzed to model people with 3D stick figures (e.g., skeletons) so that poses may be identified. However, extracting the 3D stick figures is non-trivial, and can fail in a cluttered environment due to occlusions. Moreover, the 3D stick figures do not include the context of actions, such as an object that a human subject is handling. It is therefore hard to disambiguate many different actions using the 3D stick figures alone. Also, this method is camera view dependent for successful action recognition. In this experiment, positions of the skeleton joints of each subject were normalized using the neck point and then the x-y coordinates from all four cameras were concatenated into a feature vector. A deep network was trained using a similar approach to the above ShapeContext method.

A color plus depth approach can be used as another method. This method follows the scheme of standard action recognition methods on 2D images, and is also camera view dependent. In this experiment, bounding boxes of each person were found based on tracking result. Color and depth images of each person in the video were cropped. The cropped color and depth video was used in action recognition. A deep neural network was trained using the cropped color and depth images and corresponding action labels.

PointNet is a deep learning method for object recognition and semantic segmentation on 3D point clouds. In this experiment, the model was extended to include an LSTM layer to handle sequential data for action recognition. The network was trained end-to-end using point clouds from the four camera images.

In Table <NUM> below, percentage accuracy among the competing methods are shown for each of the five groupings in the experiment. As seen in Table <NUM>, 4D dynamic solid modeling concepts (4D DSM) generally produced the highest action recognition accuracy results among the competing methods. For example, for Group <NUM>, 4D dynamic solid modeling was <NUM> percent accurate, while the next best performing method, PointNet, was only <NUM> percent accurate. Other implementations of 4D dynamic modeling and/or different comparisons may produce slightly different results, but 4D dynamic modeling can produce significantly better results than existing methods.

Table <NUM> shows how 4D dynamic solid modeling concepts can be an accurate and reliable technical solution to the technical problem of action recognition in a real-world, cluttered environment. 4D dynamic solid modeling can be invariant to camera view angles, resistant to clutter, and able to handle crowds. 4D dynamic solid modeling provides information not only about people, but also about the objects with which people are interacting. Therefore, rather than being hindered by clutter, 4D dynamic solid modeling is able to provide rich, 4D information about the complex environment.

The 4D dynamic solid modeling technical solution is able to provide rich, 4D information in real time without onerous equipment or processing resource requirements. In some cases, 4D dynamic solid modeling of an environment can be performed while using as little as <NUM> to <NUM> percent of a generic GPU. Additionally, 4D dynamic solid modeling techniques can be fast. For example, with a single GTX1080 TI, 4D dynamic solid modeling can track <NUM> people and infer their actions at <NUM> frames per second.

Stated another way, 4D dynamic solid modeling can quickly and reliably generate rich, 4D information of a complex environment, and recognize actions of tracked subjects in real time. Experimental results confirm that 4D dynamic solid modeling offers improved action recognition performance among existing computer vision methods. Even in large-scale settings, 4D dynamic solid modeling can be deployed to enhance how people interact with the environment.

<FIG> shows a system <NUM> that can accomplish 4D dynamic solid modeling concepts. For purposes of explanation, system <NUM> includes cameras <NUM>. System <NUM> also includes a controller <NUM>. The controller <NUM> can coordinate function of and/or receive data from the cameras <NUM> and/or from other sensors. System <NUM> can also include one or more devices <NUM>. In the illustrated example, device <NUM>(<NUM>) is manifest as a notebook computer device and example device <NUM>(<NUM>) is manifest as a server device. In this case, the controller <NUM> is freestanding. In other implementations, the controller <NUM> can be incorporated into device <NUM>(<NUM>). The cameras <NUM>, controller <NUM>, and/or devices <NUM> can communicate via one or more networks (represented by lightning bolts <NUM>) and/or can access the Internet over the networks. Various networks are shown in <FIG>, additional networks are contemplated. For example, in some cases cameras <NUM> could communicate with device <NUM>(<NUM>).

As illustrated relative to <FIG>, the cameras <NUM> can be proximate to an environment to which 4D dynamic solid modeling concepts are applied. Controller <NUM> and/or devices <NUM> can be proximate to the environment or remotely located. For instance, in one configuration, device <NUM>(<NUM>) could be located proximate to the environment (e.g., in the same building), while device <NUM>(<NUM>) is remote, such as in a server farm (e.g., cloud-based resource).

<FIG> shows two device configurations <NUM> that can be employed by devices <NUM>. Individual devices <NUM> can employ either of configurations <NUM>(<NUM>) or <NUM>(<NUM>), or an alternate configuration. (Due to space constraints on the drawing page, one instance of each configuration is illustrated rather than illustrating the device configurations relative to each device <NUM>). Briefly, device configuration <NUM>(<NUM>) represents an operating system (OS) centric configuration. Configuration <NUM>(<NUM>) represents a system on a chip (SOC) configuration. Configuration <NUM>(<NUM>) is organized into one or more applications <NUM>, operating system <NUM>, and hardware <NUM>. Configuration <NUM>(<NUM>) is organized into shared resources <NUM>, dedicated resources <NUM>, and an interface <NUM> there between.

In either configuration <NUM>, the device can include storage/memory <NUM>, a processor <NUM>, and/or a 4D dynamic solid modeling (4D DSM) component <NUM>. In some implementations, the 4D dynamic solid modeling component <NUM> can include a 3D solid volume construction algorithm, a subject candidate detection algorithm, a subject classifier algorithm, a subject tracking algorithm, and/or an action recognition algorithm. The 3D solid volume construction algorithm can determine voxel occupancy and/or construct a 3D solid volume representation of the environment. The subject candidate detection algorithm can find subject candidates and/or determine partial volumes within the 3D solid volume representation. The subject classifier algorithm can classify whether the partial volumes contain a subject of interest. The tracking algorithm can track subjects of interest using 4D data. The action recognition algorithm can recognize actions, interactions, and/or behaviors of the tracked subjects.

In some configurations, each of devices <NUM> can have an instance of the 4D dynamic solid modeling component <NUM>. However, the functionalities that can be performed by 4D dynamic solid modeling component <NUM> may be the same or they may be different from one another. For instance, in some cases, each device's 4D dynamic solid modeling component <NUM> can be robust and provide all the functionality described above and below (e.g., a device-centric implementation). In other cases, some devices can employ a less robust instance of the 4D dynamic solid modeling component <NUM> that relies on some functionality to be performed remotely. For instance, device <NUM>(<NUM>) has more processing resources than device <NUM>(<NUM>). In such a configuration, depth data from cameras <NUM> may be sent to device <NUM>(<NUM>). This device can use the depth data to train one or more of the algorithms introduced above. The algorithms can be communicated to device <NUM>(<NUM>) for use by 4D dynamic solid modeling component <NUM>(<NUM>). Then 4D dynamic solid modeling component <NUM>(<NUM>) can operate the algorithms in real time on data from cameras <NUM> to recognize an action of a person. In another case, the subject tracking algorithm can be accomplished by 4D dynamic solid modeling component <NUM>(<NUM>) on device <NUM>(<NUM>), while the action recognition algorithm can be accomplished by 4D dynamic solid modeling component <NUM>(<NUM>) on device <NUM>(<NUM>), for example.

The term "device", "computer", or "computing device" as used herein can mean any type of device that has some amount of processing capability and/or storage capability. Processing capability can be provided by one or more processors that can execute data in the form of computer-readable instructions to provide a functionality. Data, such as computer-readable instructions and/or user-related data, can be stored on storage, such as storage that can be internal or external to the device. The storage can include any one or more of volatile or non-volatile memory, hard drives, flash storage devices, and/or optical storage devices (e.g., CDs, DVDs etc.), remote storage (e.g., cloud-based storage), among others. As used herein, the term "computer-readable media" can include signals. In contrast, the term "computer-readable storage media" excludes signals. Computer-readable storage media includes "computer-readable storage devices". Examples of computer-readable storage devices include volatile storage media, such as RAM, and non-volatile storage media, such as hard drives, optical discs, and flash memory, among others.

Examples of devices <NUM> can include traditional computing devices, such as personal computers, desktop computers, servers, notebook computers, cell phones, smart phones, personal digital assistants, pad type computers, mobile computers, appliances, smart devices, IoT devices, etc. and/or any of a myriad of ever-evolving or yet to be developed types of computing devices.

As mentioned above, configuration <NUM>(<NUM>) can be thought of as a system on a chip (SOC) type design. In such a case, functionality provided by the device can be integrated on a single SOC or multiple coupled SOCs. One or more processors <NUM> can be configured to coordinate with shared resources <NUM>, such as memory/storage <NUM>, etc., and/or one or more dedicated resources <NUM>, such as hardware blocks configured to perform certain specific functionality. Thus, the term "processor" as used herein can also refer to central processing units (CPUs), graphical processing units (GPUs), field programmable gate arrays (FPGAs), controllers, microcontrollers, processor cores, or other types of processing devices.

Claim 1:
A system (<NUM>), comprising:
multiple 3D cameras (<NUM>) positioned relative to an environment (<NUM>) to sense the environment from different viewpoints;
a processing device (<NUM>); and
a storage device (<NUM>) storing computer-executable instructions which, when executed by the processing device, cause the processing device to:
receive (<NUM>) depth data sensed by the multiple 3D cameras over time,
determine (<NUM>) voxel occupancy of the environment by combining the depth data collected from the multiple 3D cameras, wherein the voxel occupancy is determined based on calibrated red green blue depth, RGBD, images for each camera, and taking the intersection of the voxel occupancies of the multiple 3D cameras,
construct (<NUM>) a 3D solid volume representation using the voxel occupancy wherein the 3D solid volume representation is a 3D map of voxel occupancy,
select (<NUM>) a subject using the 3D solid volume representation comprising, detecting a subject candidate by locating a local maximum with a given width and height on a Gaussian filtered, top-down image of the 3D solid volume representation,
establishing a partial volume around the subject candidate, wherein the partial volume is a portion of the 3D solid volume representation, and
using a trained classifying model to classify whether the partial volume contains a subject,
track (<NUM>) the selected subject using the depth data over time,
recognize (<NUM>) an action of the tracked subject by applying a trained deep convolutional neural network to a sequence of partial volumes of the tracked subject, and
output (<NUM>) the recognized action.