Method and system for generating a synthetic database of postures and gestures

Methods and systems for generating synthetic samples of postures and gestures are provided. In one embodiment, the system may include: a database configured to store at least one sample of a posture or a gesture; a sensing device; and a computer processor configured to: derive values of parameters relating to a specific user and/or environment and generate datasets of gesture and posture samples based on the derived values. In another embodiment, a computer memory may be configured to: obtain at least one gesture or posture of at least one body part; and obtain data sets affecting the appearance of said postures and gestures; and a computer processor configured to generate sets of synthetic samples of said postures and gestures, based on the obtained data sets, wherein the synthetic samples embody virtual variations caused by at least one of the obtained data sets.

FIELD OF THE INVENTION

The present invention relates generally to natural user interface (NUI), and more particularly to synthetically generating a database of samples that are sufficient for training a NUI system.

BACKGROUND OF THE INVENTION

Natural User interface (NUI) devices have become ever more popular with gesture capturing systems of various technologies being used to recognize gestures and postures of body parts of users.

Generally speaking, two types of NUI devices are currently available. The first type is configured for operating with many users and it would be advantageous to be able to train NUI devices of the first type to be able to operate properly with as much users as may be possible. The second type of NUI devices are configured to only work with a single user but it is unknown in advance which user will eventually use the NUI device.

Both of the aforementioned types of NUI devices lead to a training challenge: the first type poses a generalization challenge, while the second type poses a challenge of specialization.

SUMMARY OF THE INVENTION

Some embodiments of the present invention provide a method and a system for generating a large database of samples suitable for training natural user interface (NUI) of various types. Specifically, the database may store samples for training gesture recognition, hand detection, 3D hand modeling, face recognition, and full body recognition. This is achieved by using a graphics system that can render realistic simulations of human motion.

Thus, one aspect of the present invention provides the generalization of potential users of a NUI system, possibly carried out at the manufacturer of the NUI device so that the model will be able to address as many variants as possible. The other aspect is specialization of the NUI device, to be carried out at the user side, and to be as tailored as possible to a specific user.

According to some embodiments of the present invention, a system architect determines what postures and or gestures need to be supported. At that stage, the architect also determines either himself or by obtaining automatically which parameters to vary for (usually from a list). The system architect then collects or gets from a database values of the parameters for all scenarios. The architect then may receive at least one posture or gesture, either by capturing in real life or creating a simulation of it. Then, after feeding all the required data, the method in accordance with embodiments of the present invention generates a database based on required combination of parameters.

The aforementioned process may be carried out possibly where NUI system is manufactured, and the following stages may be implemented in order to achieve it: obtaining at least one gesture or posture of at least one body part of one or more persons; obtaining data sets affecting the appearance of said postures and gestures; and generating sets of synthetic samples of the postures and gestures, based on the obtained data sets. It is noted that the synthetic samples embody virtual variations caused by at least one of the obtained data sets.

In a different embodiment, and in an operation phase, while the NUI system may be already in the hands of the end user, the database may be rendered and customized on a per-user and/or per-environment basis. In order to achieve a quick “fine tuning” of the NUI system, the user may be required to provide a capturing of his or her relevant body parts (participating in the gestures and postures) and the database may be generated automatically for the specified gestures and postures, and for the specified camera and measured lighting conditions, in a quick manner.

Advantageously, since the samples on the database are all synthetically generated, each and every sample of a captured gesture or motion, is associated with exact geometric arrangement of the modeled body part, the texture, skin tone, lighting condition and actually any parameter that affect the capturing. Additionally and preferably, the synthetically generated samples will have a typical noise applied to them. In a case that the capturing system generates a depth map, the Z-buffer serves as the framework for the samples, with all of the aforementioned advantages of knowledge of the parameters related to the exact motion, distance of the body part, texture, and lighting conditions.

DETAILED DESCRIPTION OF THE INVENTION

In one embodiment, a system for generating a database of synthetic samples of postures and gestures is provided herein. The system may include a memory configured to store at least one gesture or posture of at least one body part of one or more persons; and a computer processor configured to: determine use case-specific values of parameters of factors affecting the appearance of said postures and gestures; and generate use case-specific synthetic samples of said postures and gestures, based on the obtained use case-specific values of parameters, wherein the synthetic samples embody virtual variations caused by at least one of the obtained use case-specific values.

FIG. 1is a block diagram illustrating non-limiting exemplary architecture of a system for generating a generalized database of synthetic samples of postures and gestures in accordance with some embodiments of the present invention. System100may include a computer processor130connected to a memory140on which sets of instructions may be loaded to be executed by computer processor130. System100may include several software modules that may be executed by computer processor130and possibly by an optional graphical processing unit (GNU)135. Such software modules may include (but are not limited to) a postures and gestures generator120, a lighting module150, a texture module160, an appearance module170, and a motion module180.

In operation, memory140may be configured to obtain at least one gesture or posture of at least one body part of one or more persons (not shown here). Memory140may be further configured to obtain data sets affecting the appearance of the postures and gestures.

Then, responsive to selection of parameters, possibly by an architect10and over a user interface110, computer processor130, using postures and gestures generator120which in turn may use lighting module150, texture module160, appearance module170, and motion module180, may be configured to generate sets of synthetic samples of said postures and gestures, based on the obtained data sets to be stored on synthetic postures and gestures database190.

Specifically, the synthetic samples stored on synthetic postures and gestures database190may be configured to embody virtual variations caused by at least one of the obtained data sets.

In accordance with some embodiments of the present invention, the generation of the samples is further based on real-life samples195(e.g., of actual people captured waving their hands for example). Alternatively, no real-life samples may be required and the synthetic samples can be generated in a full synthetic manner.

Specifically, lighting module150may be configured to provide varying lighting conditions, texture module160may be configured to provide varying textures of human users and/or clothing items, appearance module170may be configured to provide varying sizes, shapes, and colors of body parts and/or clothing items, whereas motion module180may be configured to provide varying motion parameters such as speed and acceleration of body parts of users.

According to some embodiments of the present invention, the synthetic database190may contain several types of valid motions which are distinguishable from each other, types of motions that should be ignored as they are not valid (e.g., they do not translate to a meaningful action based on the gesture/posture recognition system) as well as some other non-meaningful “noise” samples that serve as negative examples.

In accordance with other embodiments of the present invention, the database is generated using different configurations of cameras—based on various parameters that define the camera properties such as field of view (FOV), resolution, sensor size, and the like.

FIG. 2is a high level flowchart illustrating non-limiting exemplary method of generating a database of synthetic samples of postures and gestures, in accordance with embodiments of the present invention. Method200may include the following steps: obtaining at least one gesture or posture of at least one body part of one or more persons210; obtaining data sets affecting the appearance of said postures and gestures220; and generating sets of synthetic samples of said postures and gestures, based on the obtained data sets230, wherein the synthetic samples embody virtual variations caused by at least one of the obtained data sets.

FIG. 3is a block diagram illustrating non-limiting exemplary architecture of a system for rendering use case specific data sets of synthetic samples of postures and gestures, in accordance with some embodiments of the present invention. System300may include a database390configured to store at least one sample of a posture or a gesture and may possibly be either located locally or located in a remote location and may communicate over a network40. System300may also include a sensing device50(which can be a webcam connected to a PC); and a computer processor310. System300may include several software modules that may be executed by computer processor310. Such software modules may include: a lighting module350, a texture module360, an appearance module370, and a motion module380.

In operation, computer processor310may be configured to derive values of parameters relating to a specific user, wherein the derived parameters are predefined upon generating database390(possibly similar to database190described herein in reference toFIG. 1but more preferably merely a subset thereof that is sufficient to generate some synthetic samples but not all possible variants as database190). Computer processor310may be further configured to generate datasets320of gesture and posture samples based on the derived values, possibly by using lighting module350, texture module360, appearance module370, and motion module380.

According to some embodiments of the present invention, system300may further include a user interface60configured to monitor a sampling process of data of a user by sensing device50.

As explained above, and according to some embodiments of the present invention, database390may be connected remotely to the computer processor via a network40. The sensing device50may be a webcam operated via a PC of user20. Thus, in a minimal implementation, the actual device near user20include only the computer processor and the generated datasets, which in some embodiments can also be stored remotely and accessed via a network (e.g., via a cloud). In yet another embodiment of the invention, the computer processor that carries out the adaptation to a specific device may be located remotely (e.g., on a cloud), and the resultant database or classifier produced by the computer processor remotely is then transferred to a local device. Alternatively, it can also be sent to a different cloud in a case further processing takes place there.

In accordance with some other embodiments of the present invention, a dataset may be generated on a per-user and/or per-ambience basis. A hand or another body part of a user may be captured for deriving the texture of the body part participating in the posture or the gestures. Then, the system automatically generates the samples of the gestures and the postures and uses the captured texture implemented on the synthetic postures and gestures. It should be noted therefore that the textures are not related to the postures or the gestures themselves, but rather to the body that is performing the gesture.

FIG. 4is a high level flowchart illustrating the method of rendering datasets of synthetic samples of postures and gestures for a specific user, in accordance with embodiments of the present invention. Method400may include the following steps: accessing a database comprising a plurality of posture and gesture samples indexed (or possibly clustered) based on parameters that are predefined upon generating said database410; obtaining, via a sensing device, samples of a user420; deriving values of the parameters relating to a specific user, based on the samples430; and generating datasets of gesture and posture samples based on the derived parameters440.

According to some embodiments of the present invention, some of the parameters that are related to the end-user for the database generated either at the factory (generalized database) or as the end user (scenario-specific, at the end user) may be user-dependent and may include at least some of: texture of the bodily part which may relate to: skin tone, hair, clothing (e.g., sleeves, gloves).

Additionally, the overall appearance may be parameterized into, for example: shape of the body part, dimensions (length (e.g., of arm, of finger), and width), motion speed (e.g., in case the gesture is “waving”—the rate or frequency of the waving); motion “arc” (i.e., a specific gesture does not have to have a full path, but start late and/or end early such as in a “Hello” gesture, waving the arm left to right. In a base gesture, it goes from 45 degrees to the left to 45 degrees to the right. But it can be from −30° to +30°, or −45° to +30°, or −30° to +45°, and the like).

Additionally, environment conditions can also be parameterized using at least some of the following metrics: lighting conditions which include light color (yellow, white, and the like) but also the shape of the light source such as single light source, light-bulb, florescent, ambient light, projectors, and the like. The environment also includes backgrounds in particular static backgrounds (horizon, single color wall, colored wall, furniture, and the like) but also moving backgrounds (dynamic scene, capturing during motion, crowd of people).

In addition, the sensor conditions may also be parameterized based on at least some of the following metrics: sensor type (camera, depth sensor, other sensing devices), wherein for cameras the following parameters can be used: sensor wavelength range such as visible light, monochrome, Infra-Red (IR); rolling/global shutter; sensor resolution; and field of view (FOV).

According to some embodiments of the present invention, at the factory side where the system is being manufactured, an architect may configure the database to encompass a subset of the possible database by the following process: determine what set of gestures and/or postures the database should contain (if for training, determine a set of positive and/or negative example); determine which parameters that effect the capture of the poses/gestures (from list above); determine possible values/value-ranges for such parameters, e.g.: samples of skin-tones and/or clothing types (rendered and/or sampled from actual users); Typical lighting conditions.

The architect then can go on to determine the set of combinations of parameters to employ. This could be the full set (all combinations) or a subset determined either manually, randomly, or through some decision process, automated or manual (e.g., which is meant to make sure every specific parameter is represented at least N times, such as N=3).

The architect then can go on to determine for each posture/gesture a “base” set, which could either be: collected from one or more individuals, one or more Synthetically created (in a simulation and/or modelling software) or a combination thereof. It should be noted that in some embodiments no real world samples are actually needed.

The architect then can go on to determine rendering using a common or proprietary or special developed rendering engine, variations of the pose and/or gesture, by adjusting for each one of the combinations determined previously.

Then, in operation, the aforementioned database can be used either for training a classifier offline, and applying it online. Additionally or alternatively the database can be used for classification online, by using sorts of nearest-neighbor classification such as by measuring distances in an N-dimensional space.

In accordance with some embodiments of the present invention, in an exemplary non-limiting use case, computer processor310may be further configured to generate an avatar based on samples obtained (possibly via sensing unit50) from one end user (e.g., user20) by applying a sequence of postures and/or gestures derived from a second end-user (possibly a sequence that describes a specified dance) stored on synthetic database390. Thus, the generated avatar may include, for example, a synthetic image of a dancing figure having the mobility characteristics of the second user and the appearance of a first user (e.g. user20).

In order to implement methods200and400according to some embodiments of the present invention, a computer processor may receive instructions and data from a read-only memory or a random access memory or both. At least one of aforementioned steps is performed by at least one processor associated with a computer. The essential elements of a computer are a processor for executing instructions and one or more memories for storing instructions and data. Generally, a computer will also include, or be operatively coupled to communicate with, one or more mass storage devices for storing data files. Storage modules suitable for tangibly embodying computer program instructions and data include all forms of non-volatile memory, including by way of example semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices and also magneto-optic storage devices.

Some aspects of the present invention are described above with reference to flowchart illustrations and/or portion diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each portion of the flowchart illustrations and/or portion diagrams, and combinations of portions in the flowchart illustrations and/or portion diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or portion diagram portion or portions.

The aforementioned flowchart and diagrams illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each portion in the flowchart or portion diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the portion may occur out of the order noted in the figures. For example, two portions shown in succession may, in fact, be executed substantially concurrently, or the portions may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each portion of the portion diagrams and/or flowchart illustration, and combinations of portions in the portion diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

Any publications, including patents, patent applications and articles, referenced or mentioned in this specification are herein incorporated in their entirety into the specification, to the same extent as if each individual publication was specifically and individually indicated to be incorporated herein. In addition, citation or identification of any reference in the description of some embodiments of the invention shall not be construed as an admission that such reference is available as prior art to the present invention.