ESTIMATION MODEL FOR INTERACTION DETECTION BY A DEVICE

A method and device are disclosed for estimating an interaction with the device. The method includes configuring a first token and a second token of an estimation model according to first features of a 3D object, applying a first weight to the first token to produce a first-weighted input token and applying a second weight that is different from the first weight to the second token to produce a second-weighted input token, and generating, by a first encoder layer of an estimation-model encoder of the estimation model, an output token based on the first-weighted input token and the second-weighted input token. The method may include receiving, at a 2D feature extraction model, the first features from a backbone, extracting, by the 2D feature extraction model, second features including 2D features, and receiving, at the estimation-model encoder, data generated based on the 2D features.

TECHNICAL FIELD

The disclosure generally relates to machine learning. More particularly, the subject matter disclosed herein relates to improvements to the detection of interactions with a device using machine learning.

SUMMARY

Devices (e.g., virtual reality (VR) devices, augmented reality (AR) devices, communications devices, medical devices, appliances, machines, etc.) may be configured to make determinations about interactions with the devices. For example, a VR or AR device may be configured to detect human-device interactions, such as specific hand gestures (or hand poses). The device may use information associated with the interactions to perform an operation on the device (e.g., changing a setting on the device). Similarly, any device may be configured to estimate different interactions with the device and perform operations associated with the estimated interactions.

To solve the problem of accurately detecting interactions with a device, a variety of machine learning (ML) models have been applied. For example, convolutional neural network- (CNN-) based models and transformer-based models have been applied.

One issue with the above approaches is that the accuracy of estimating interactions (e.g., hand poses) may be reduced in some situations due to self-occlusion, camera distortion, three-dimensional (3D) ambiguity of projection, etc. For example, self-occlusion may commonly occur in hand-pose estimation, where one part of a user's hand may be occluded by (e.g., covered by) another part of the user's hand from the viewpoint of the device. Thus, the accuracy of estimating the hand pose and/or distinguishing between similar hand gestures may be reduced.

To overcome these issues, systems and methods are described herein for improving an accuracy of a device to estimate interactions with the device by using a machine learning model with a pre-sharing mechanism, two-dimensional (2D) feature map extraction, and/or a dynamic-mask mechanism.

The above approaches improve on previous methods because accuracy may be improved, and better performance may be achieved in mobile devices having limited computing resources.

Some embodiments of the present disclosure provide for a method for using an estimation model having 2D feature extraction between a backbone and an estimation-model encoder.

Some embodiments of the present disclosure provide for a method for using an estimation model having pre-sharing weights in an encoder layer of a Bidirectional Encoder Representations from Transformers (BERT) encoder of the estimation-model encoder.

Some embodiments of the present disclosure provide for a method for using an estimation model having 3D hand joints and mesh points estimated by applying camera intrinsic parameters to one or more BERT encoders, along with hand tokens from a previous BERT encoder, as inputs to the one or more BERT encoders. For example, camera intrinsic parameters may be applied, along with hand tokens from a fourth BERT encoder, as inputs to a fifth BERT encoder. While embodiments involving hands and hand joints are discussed herein, it will be appreciated that the embodiments and techniques described are applicable, without limit, to any mesh or model, including those of various other body parts.

Some embodiments of the present disclosure provide for a method for using an estimation model with data generated based on 2D feature map.

Some embodiments of the present disclosure provide for a method for using an estimation model with a dynamic-mask mechanism.

Some embodiments of the present disclosure provide for a method for using an estimation model trained with a data set generated based on 2D-image rotation and rescaling that is projected to 3D in an augmentation process.

Some embodiments of the present disclosure provide for a method for using an estimation model trained with two optimizers.

Some embodiments of the present disclosure provide for a method for using an estimation model having BERT encoders with more than four (e.g., twelve) encoder layers.

Some embodiments of the present disclosure provide for a method for using an estimation model having hyper-parameters that are mobile-friendly by using fewer transformers or smaller transformers in each BERT encoder than would be used in a large device with more computing resources.

Some embodiments of the present disclosure provide for a device on which an estimation model may be implemented.

According to some embodiments of the present disclosure, a method of estimating an interaction with a device includes configuring a first token and a second token of an estimation model according to one or more first features of a 3-dimensional (3D) object, applying a first weight to the first token to produce a first-weighted input token and applying a second weight that is different from the first weight to the second token to produce a second-weighted input token, and generating, by a first encoder layer of an estimation-model encoder of the estimation model, an output token based on the first-weighted input token and the second-weighted input token.

The method may further include receiving, at a backbone of the estimation model, input data corresponding to the interaction with the device, extracting, by the backbone, the one or more first features from the input data, receiving, at a two-dimensional (2D) feature extraction model, the one or more first features from the backbone, extracting, by the 2D feature extraction model, one or more second features associated with the one or more first features, the one or more second features including one or more 2D features, receiving, at the estimation-model encoder, data generated based on the one or more 2D features, generating, by the estimation model, an estimated output based on the output token and the data generated based on the one or more 2D features, and performing an operation based on the estimated output.

The data generated based on the one or more 2D features may include an attention mask.

The first encoder layer of the estimation-model encoder may correspond to a first BERT encoder of the estimation-model encoder, and the method may further include concatenating a token, associated with an output of the first BERT encoder, with at least one of camera intrinsic-parameter data, three-dimensional (3D) hand-wrist data, or bone-length data to generate concatenated data, and receiving the concatenated data at a second BERT encoder.

The first BERT encoder and the second BERT encoder may be included in a chain of BERT encoders, the first BERT encoder and the second BERT encoder may be separated by at least three BERT encoders of the chain of BERT encoders, and the chain of BERT encoders may include at least one BERT encoder having more than four encoder layers.

A data set used to train the estimation model may be generated based on two-dimensional (2D) image rotation and rescaling that is projected to three dimensions (3D) in an augmentation process, and a backbone of the estimation model may be trained using two optimizers.

The device may be a mobile device, the interaction may be a hand pose, and the estimation model may include hyperparameters including at least one of an input feature dimension that is about equal to 1003/256/128/32 for estimating 195 hand-mesh points, an input feature dimension that is about equal to 2029/256/128/64/32/16 for estimating 21 hand joints, a hidden feature dimension that is about equal to 512/128/64/16 (4H, 4L) for estimating 195 hand-mesh points, or a hidden feature dimension that is about equal to 512/256/128/64/32/16 (4H, (1, 1, 1, 2, 2, 2)L) for estimating 21 hand joints.

The method may further include generating a 3D scene including a visual representation of the 3D object, and updating the visual representation of the 3D object based on the output token.

According to other embodiments of the present disclosure, a method of estimating an interaction with a device includes receiving, at a two-dimensional (2D) feature extraction model of an estimation model, one or more first features corresponding to input data associated with an interaction with the device, extracting, by the 2D feature extraction model, one or more second features associated with the one or more first features, the one or more second features including one or more 2D features, generating, by the 2D feature extraction model, data based on the one or more 2D features, and providing the data to an estimation-model encoder of the estimation model.

The method may further include receiving, at a backbone of the estimation model, the input data, generating, by the backbone, the one or more first features based on the input data, associating a first token and a second token of the estimation model with the one or more first features, applying a first weight to the first token to produce a first-weighted input token and applying a second weight that is different from the first weight to the second token to produce a second-weighted input token, calculating, by a first encoder layer of the estimation-model encoder, an output token based on receiving the first-weighted input token and the second-weighted input token as inputs, and generating, by the estimation model, an estimated output based on the output token and the data generated based on the one or more 2D features, and performing an operation based on the estimated output.

The data generated based on the one or more 2D features may include an attention mask.

The estimation-model encoder may include a first BERT encoder including a first encoder layer, and the method may further include concatenating a token, corresponding to an output of the first BERT encoder, with at least one of camera intrinsic-parameter data, three-dimensional (3D) hand-wrist data, or bone-length data to generate concatenated data, and receiving the concatenated data at a second BERT encoder.

The first BERT encoder and the second BERT encoder may be included in a chain of BERT encoders, the first BERT encoder and the second BERT encoder may be separated by at least three BERT encoders of the chain of BERT encoders, and the chain of BERT encoder may include at least one BERT encoder having more than four encoder layers.

A data set used to train the estimation model may be generated based on 2D-image rotation and rescaling that is projected to three dimensions (3D) in an augmentation process, and a backbone of the estimation model may be trained using two optimizers.

The device may be a mobile device, the interaction may be a hand pose, and the estimation model may include hyperparameters including at least one of an input feature dimension that is about equal to 1003/256/128/32 for estimating 195 hand-mesh points, an input feature dimension that is about equal to 2029/256/128/64/32/16 for estimating 21 hand joints, a hidden feature dimension that is about equal to 512/128/64/16 (4H, 4L) for estimating 195 hand-mesh points, or a hidden feature dimension that is about equal to 512/256/128/64/32/16 (4H, (1, 1, 1, 2, 2, 2)L) for estimating 21 hand joints.

The method may further include calculating, by a first encoder layer of the estimation-model encoder, an output token, generating a 3D scene including a visual representation of the interaction with the device, and updating the visual representation of the interaction with the device based on the output token.

According to other embodiments of the present disclosure, a device configured to estimate an interaction with the device includes a memory, and a processor communicably coupled to the memory, wherein the processor is configured to receive, at a two-dimensional (2D) feature extraction model of an estimation model, one or more first features corresponding to input data associated with an interaction with the device, generate, by the 2D feature extraction model, one or more second features based on the one or more first features, the one or more second features including one or more 2D features, and send, by the 2D feature extraction model, data generated based on the one or more 2D features to an estimation-model encoder of the estimation model.

The processor may be configured to receive, at a backbone of the estimation model, the input data, generate, by the backbone, the one or more first features based on the input data, associate a first token and a second token of the estimation model with the one or more first features, apply a first weight to the first token to produce a first-weighted input token and applying a second weight that is different from the first weight to the second token to produce a second-weighted input token, calculate, by a first encoder layer of the estimation-model encoder, an output token based on receiving the first-weighted input token and the second-weighted input token as inputs, generate, by the estimation model, an estimated output based on the output token and the data generated based on the one or more 2D features, and perform an operation based on the estimated output.

The data generated based on the one or more 2D features may include an attention mask.

The estimation-model encoder may include a first BERT encoder including a first encoder layer, and the processor may be configured to concatenate a token, corresponding to an output of the first BERT encoder, with at least one of camera intrinsic-parameter data, three-dimensional (3D) hand-wrist data, or bone-length data to generate concatenated data, and receive the concatenated data at a second BERT encoder.

DETAILED DESCRIPTION

FIG.1is block diagram depicting a system including an estimation model, according to some embodiments of the present disclosure.

Aspects of embodiments of the present disclosure may be used in augmented reality (AR) or virtual reality (VR) devices for high-accuracy 3D hand-pose estimation from a single camera so as to provide hand pose information in human-device interaction processes. Aspects of embodiments of the present disclosure may provide for accurate hand-pose estimation including 21 hand joints and hand meshes in 3D from a single RGB image and in real-time for human-device interaction.

Referring toFIG.1, a system1for estimating an interaction with a device100may involve determining a hand pose (e.g., a 3D hand pose) based on analyzing image data2(e.g., image data associated with a 2D image). The system1may include a camera10for capturing image data2associated with the interaction with the device100. The system1may include a processor104(e.g., a processing circuit) communicably coupled with a memory102. The processor104may include a central processing unit (CPU), a graphics processing unit (GPU), and/or a neural processing unit (NPU). The memory102may store weights and other data for processing input data12to generate estimated outputs32from an estimation model101(e.g., a hand-pose estimation model). The input data12may include the image data2, 3D hand-wrist data4, bone-length data5(e.g., bone length), and camera intrinsic-parameter data6. For example, the camera intrinsic-parameter data6may include information that is intrinsic to the camera10, such as the location of the camera, focal-length data, and/or the like. Although the present disclosure describes hand-pose estimation, it should be understood that the present disclosure is not limited thereto and that the present disclosure may be applied to estimating a variety of interactions with the device100. For example, the present disclosure may be applied to estimating any part of, or an entirety of, any object that may interact with the device100.

The device100may correspond to the electronic device601ofFIG.6. The camera10may correspond to the camera module680ofFIG.6. The processor104may correspond to the processor620ofFIG.6. The memory102may correspond to the memory630ofFIG.6.

Referring still toFIG.1, the estimation model101may include a backbone110, an estimation-model encoder120, and/or a 2D feature-extraction model115, all of which are discussed in further detail below. The estimation model101may include software components stored on the memory102and processed with the processor104. A “backbone” as used herein refers to a neural network that has been previously trained in other tasks and, thus, has a demonstrated effectiveness for estimating (e.g., predicting) outputs based on a variety of inputs. Some examples of a “backbone” are CNN-based or transformer-based backbones, such as a visual attention network (VAN) or a high-resolution network (HRNet). The estimation-model encoder120may include one or more BERT encoders (e.g., a first BERT encoder201and a fifth BERT encoder205). A “BERT encoder” as used herein refers to a machine learning structure that includes transformers to learn contextual relationships between inputs. Each BERT encoder may include one or more BERT encoder layers (e.g., a first encoder layer301). An “encoder layer” (also referred to as a “BERT encoder layer” or a “transformer”) as used herein refers to an encoder layer having an attention mechanism, which uses tokens as a basic unit of input to learn and predict high-level information from all tokens based on the attention (or relevance) of each individual token compared with all tokens. An “attention mechanism” as used herein refers to a mechanism that enables a neural network to concentrate on some portions of input data while ignoring other portions of the input data. A “token” as used herein refers to a data structure used to represent one or more features extracted from input data, wherein the input data includes positional information.

The 2D feature-extraction model115may be located between the backbone110and the estimation-model encoder120. The 2D feature-extraction model115may extract 2D features associated with the input data12. The 2D feature-extraction model115may provide data generated based on the 2D features to the estimation-model encoder120to improve the accuracy of the estimation-model encoder120, as is discussed in further detail below.

The estimation model101may process the input data12to generate an estimated output32. The estimated output32may include a first estimation-model output32aand/or a second estimation-model output32b. For example, the first estimation-model output32amay include an estimated 3D hand-joint output, and the second estimation-model output32bmay include an estimated 3D hand-mesh output. In some embodiments, the estimated output32may include 21 hand joints and/or a hand mesh with 778 vertices in 3D. The device100may use the estimated 3D hand-joint output to perform an operation associated with a gesture corresponding to the estimated 3D hand-joint output. The device100may use the estimated 3D hand-mesh output to present a user of the device with a virtual representation of the user's hand.

In some embodiments, the device100may generate a 3D scene including a visual representation of a 3D object (e.g., the estimated 3D hand-joint output and/or the estimated 3D hand-mesh output). The device100may update the visual representation of the 3D object based on an output token (seeFIG.5Aand the corresponding description below).

In some embodiments, the estimation model101may be trained using two optimizers to improve accuracy. For example, the training optimizers may include Adam with weight decay (AdamW) and stochastic gradient descent with weight decay (SGDW). In some embodiments, the estimation model101may be trained with GPUs for AR and/or VR device applications.

In some embodiments, a data set used to train the estimation model101may be generated based on 2D-image rotation and rescaling that is projected to 3D in an augmentation process, which may improve the robustness of the estimation model101. That is, the data set used for training may be generated using 3D-perspective hand-joint augmentation or 3D-perspective hand-mesh augmentation.

In some embodiments, the estimation model101may be configured to be mobile friendly by using parameters (e.g., hyper-parameters, including input feature dimensions and/or hidden feature dimensions) to provide real-time model performance (e.g., greater than 30 frames per second (FPS)) from limited computing resources. For example, each BERT encoder of the estimation-model encoder120in a mobile-friendly (or small model) design may include fewer transformers and/or smaller transformers than a large-model design, such that the small model may still achieve real-time performance with fewer computational resources.

For example, a first small-model version of an estimation model101may have the following parameters for estimating 195 hand-mesh points (or vertices). A backbone parameter size (in millions (M)) may be equal to about 4.10; an estimation-model encoder parameter size (M) may be equal to about 9.13; a total parameter size (M) may be equal to about 13.20; an input feature dimension may be equal to about 1003/256/128/32; a hidden feature dimension (head number, encoder layer number) may be equal to about 512/128/64/16 (4H, 4L); and a corresponding FPS may be equal to about 83 FPS.

A second small-model version of an estimation model101may have the following parameters for estimating 21 hand joints. A backbone parameter size (M) may be equal to about 4.10; an estimation-model encoder parameter size (M) may be equal to about 5.23; a total parameter size (M) may be equal to about 9.33; an input feature dimension may be equal to about 2029/256/128/64/32/16; a hidden feature dimension (head number, encoder layer number) may be equal to about 512/256/128/64/32/16 (4H, (1, 1, 1, 2, 2, 2)L); and a corresponding FPS may be equal to about 285 FPS.

In some embodiments, mobile-friendly versions of the estimation model101may have a reduced number of parameters and floating-point operations (FLOPs) by shrinking the number of encoder layers and applying a VAN instead of HRNet-w64 in a BERT encoder block, which enables high-accuracy hand-pose estimation in mobile devices in real time.

FIG.2Ais a block diagram depicting 2D feature map extraction, according to some embodiments of the present disclosure.

Referring toFIG.2A, the backbone110may be configured to extract (e.g., to remove or to generate data based on) features from the input data12. As discussed above, the input data12may include 2D-image data. One or more estimation-model-encoder input-preparation operations (e.g., reading or writing associated with data flows)22a-gmay be associated with the backbone output data22. For example, global features GF from the backbone output data22may be duplicated at operation22a. The backbone output data22(e.g., intermediate-output data IMO associated with the backbone output data22) may be sent to the 2D feature-extraction model115at operation22b. A 3D hand-joint template HJT and/or a 3D hand-mesh template HMT may be concatenated with the global features GF at operations22cand22dto associate one or more features from the backbone110with hand joints J and/or vertices V. As discussed in further detail below with respect toFIG.2B, the 2D feature-extraction model115may extract 2D features from the intermediate-output data IMO and send data generated based on the 2D features to be concatenated with the global features GF and with the 3D hand-joint template HJT and/or the 3D hand-mesh template HMT at operations22e1and22e2. The 2D feature-extraction model115may also send data generated based on the 2D features to the estimation-model encoder120at operation22e3.

FIG.2Bis a block diagram depicting a 2D feature-extraction model, according to some embodiments of the present disclosure.

Referring toFIG.2B, the intermediate-output data IMO may be received by the 2D feature-extraction model115. The intermediate-output data IMO may be provided as an input to 2D convolutional layers30and to an interpolation layer function33. The 2D convolutional layers30may output a predicted attention mask PAM and/or predicted 2D hand-joint or 2D hand-mesh data31. The predicted 2D hand-joint or 2D hand-mesh data31may be sent for concatenation at operations22e1and22e2(seeFIG.2A). The predicted attention mask PAM may be sent to the estimation-model encoder120at operation22e3(seeFIG.2A).

FIG.3is a block diagram depicting a structure of an estimation-model encoder of the estimation model, according to some embodiments of the present disclosure.

Referring toFIG.3, extracted features from the backbone output data22may be provided as inputs to a chain of BERT encoders201-205. For example, one or more features from the backbone output data22may be provided as an input to a first BERT encoder201. In some embodiments, an output of the first BERT encoder201may be provided as an input to a second BERT encoder202; an output of the second BERT encoder202may be provided as an input to a third BERT encoder203; and an output of the third BERT encoder203may be provided as an input to a fourth BERT encoder204. In some embodiments, the output of the fourth BERT encoder204may correspond to hand tokens7. The hand tokens7may be concatenated with camera intrinsic-parameter data6and/or with 3D hand-wrist data4and/or bone-length data5to generate concatenated data CD. For example, in some embodiments, the hand tokens7may be concatenated with camera intrinsic-parameter data6and with either 3D hand-wrist data4or bone-length data5. The camera intrinsic-parameter data6, the 3D hand-wrist data4, and the bong-length data5may be expanded before being concatenated with the hand tokens7. The concatenated data CD may be received as an input to a fifth BERT encoder205. The output of the fifth BERT encoder205may be split at operation28and provided to the first estimation-model output32aand to the second estimation-model output32b. Although the present disclosure refers to five BERT encoders in the chain of BERT encoders. It should be understood that the present disclosure is not limited thereto. For example, more or less than five BERT encoders may be provided in the chain of BERT encoders.

FIG.4is a block diagram depicting a structure of a BERT encoder of the estimation-model encoder, according to some embodiments of the present disclosure.

Referring toFIG.4, one or more BERT encoders of the estimation-model encoder120may include one or more encoder layers. For example, the first BERT encoder201may have L encoder layers (with L being an integer greater than zero). In some embodiments, L may be greater than four and may result in an estimation-model encoder with greater accuracy than embodiments having four or fewer encoder layers. For example, in some embodiments, L may be equal to 12.

In some embodiments, extracted features from the backbone output data22may be provided as inputs to the first encoder layer301of the first BERT encoder201. In some embodiments the extracted features from the backbone output data22may be provided as inputs to one or more linear operations LO (operations provided by linear layers) and positional encoding 251 before being provided to the first encoder layer301. An output of the first encoder layer301may be provided to an input of a second encoder layer302, and an output of the second encoder layer302may be provided to an input of a third encoder layer303. That is, the first BERT encoder201may include a chain of encoder layers with L encoder layers total. In some embodiments, an output of the L-th encoder layer may be provided as an input to one or more linear operations LO before being sent to an input of the second BERT encoder202. As discussed above, the estimation-model encoder120may include a chain of BERT encoders (e.g., including the first BERT encoder201, the second BERT encoder202, a third BERT encoder203, etc.). In some embodiments, each BERT encoder in the chain of BERT encoders may include L encoder layers.

FIG.5Ais a block diagram depicting a structure of an encoder layer of the BERT encoder with a pre-sharing weight mechanism, according to some embodiments of the present disclosure.

As an overview, the structure of the estimation-model encoder120(seeFIG.4) allows for image features to be concatenated with sampled 3D hand-pose positions and embedded with 3D position embedding. A feature map may be split into a number of tokens. Thereafter, as will be discussed in further detail below, an attention of each hand pose position may be calculated by a pre-sharing weight attention. The feature map may be extended to a number of tokens, where each token may focus on the prediction of one hand-pose position. The tokens may pass a pre-sharing weight attention layer, where the weights of an attention value layer may be different for different tokens. Hand-joint and hand-mesh tokens may then be processed by different graph convolutional networks (GCNs) (also referred to as graph convolutional neural networks (GCNNs)). Each GCN may have different weights for different tokens. The hand-joint tokens and hand-mesh tokens may be concatenated together and processed by a fully convolutional network for 3D hand pose position prediction.

Referring toFIG.5A, each encoder layer (e.g., the first encoder layer301) of each BERT encoder (e.g., the first encoder layer301) may include an attention module308(e.g., a multi-head self-attention module), residual networks340, GCN blocks350, and a feed forward network360. The attention module308may include an attention mechanism310(e.g., a scaled dot-product attention) associated with attention-mechanism inputs41-43and attention-mechanism outputs321. For example, first attention-mechanism inputs41may be associated with queries, second attention-mechanism inputs42may be associated with keys, and third attention-mechanism inputs43may be associated with values. The queries, keys, and values may be associated with input tokens. In some embodiments, and as discussed in further detail below with respect toFIGS.5B-5D, the third attention-mechanism inputs43may be associated with pre-sharing weights for improved accuracy. For example, the pre-sharing weights may correspond to weighted input tokens WIT. In some embodiments, the attention-mechanism outputs321may be provided for concatenation at operation320. An encoder layer output380of the first encoder layer301may include an output token OT. The output token OT may be calculated, by the first encoder layer301, based on receiving the weighted input tokens WIT.

The attention mechanism310may include a first multiplication function312, a second multiplication function318, a scaling function314, and a softmax function316. The first attention-mechanism inputs41and the second attention-mechanism inputs42may be provided to the first multiplication function312and the scaling function314to produce a normalized score315. The normalized score315and an attention map AM may be provided to the softmax function316to produce an attention score317. The attention score317and the third attention-mechanism inputs43may be provided to a second multiplication function318to produce an attention-mechanism output321.

FIG.5Bis a diagram depicting full sharing with a same weight applied to different input tokens of the encoder layer of the BERT encoder, according to some embodiments of the present disclosure.

FIG.5Cis a diagram depicting pre-aggregation sharing with different weights applied to different input tokens of the encoder layer of the BERT encoder, according to some embodiments of the present disclosure.

As an overview,FIG.5Cdepicts pre-sharing weights of a GCN and a value layer of an attention mechanism (also referred to as an attention block). In a pre-sharing mode, which is different from a full-sharing mode, weights for different tokens (also referred to as nodes) may be different before values are aggregated to the updated tokens (or output tokens). Therefore, different transformations may be applied to the input features of each token before they are aggregated.

FIG.5Dis a diagram depicting hand joints and a hand mesh, according to some embodiments of the present disclosure.

Referring to the hand joint HJ structure ofFIG.5D, in some embodiments, another GCN for hand-joint estimation may be added. The hand-joint structure ofFIG.5Dmay be used to generate an adjugate matrix in the GCN blocks.

Referring toFIG.5B-5D, a first input token T1and a second input token T2of the estimation-model encoder120(seeFIG.1) may be associated with one or more features extracted from the input data12(seeFIG.1). In the case of hand-pose estimation, each token may correspond to a different hand joint HJ or to a different hand-mesh vertex HMV (also referred to as a hand-mesh point).

Referring toFIG.5B, in some embodiments, the attention module308(seeFIG.5A) may not include pre-sharing. For example, in some embodiments, the attention module308may include full sharing, instead of pre-sharing (also referred to as pre-aggregation sharing as depicted inFIG.5C). In full sharing, the same weight W may be applied to each input token (e.g., the first input token T1, the second input token T2and a third input token T3) to calculate each output token (e.g., a first output token T1′, a second output token T2′, and a third output token T3′).

Referring toFIG.5C, in some embodiments, the attention module308(seeFIG.5A) may include pre-sharing. In pre-sharing, different weights may be applied to each input token. For example, a first weight Wa may be applied to the first input token T1to produce a first-weighted input token WIT1; a second weight Wb may be applied to the second input token T2to produce a second-weighted input token WIT2; and a third weight We may be applied to a third input token T3to produce a third-weighted input token WIT3. The weighted input tokens WIT may be received as inputs to the attention module308. The first encoder layer301may calculate the output token OT based on receiving the weighted input tokens WIT as inputs to the attention mechanism310. The use of different weights in pre-sharing may cause different equations to be used to estimate different hand joints (or hand mesh vertices), which may result in improved accuracy over full sharing approaches.

FIG.5Eis a block diagram depicting a dynamic mask mechanism, according to some embodiments of the present disclosure.

Referring toFIG.5E, at operation22e3(see alsoFIG.2A), the estimation-model encoder120may be provided with a predicted attention mask PAM from the 2D feature-extraction model115(seeFIG.2B). The predicted attention mask PAM may indicate which hand joints HJ or which hand-mesh vertices HMV (seeFIG.5D) are occluded (e.g., covered, obstructed, or hidden from view) in the image data2(seeFIG.1). The predicted attention mask PAM may be used by a dynamic mask-updating mechanism311of the estimation-model encoder120to update the attention map AM used by the attention mechanism310to produce a masked attention map MAM. That is, in some embodiments, the dynamic mask-updating mechanism311may be used as an extra operation, associated with the attention mechanism310discussed above with respect toFIG.5A, to improve the accuracy of the estimation model101. By updating the attention map AM to indicated occluded hand joints HJ or hand-mesh vertices HMV, an accuracy of the estimation model101may be improved because the masked tokens MT, represented in the predicted attention mask PAM and the masked attention map MAM, may reduce an amount of noise that might otherwise be associated with the occluded hand joints HJ or hand-mesh vertices HMV. In some embodiments, the masked tokens (depicted as shaded squares in the predicted attention mask PAM and in the masked attention map MAM) may be represented by very large values. The predicted attention mask PAM may be updated to produce an updated predicted attention mask PAM′ in subsequent processing cycles. For example, the predicted attention mask PAM may be updated as unobstructed (e.g., non-occluded) hand joints HJ or unobstructed hand-mesh vertices HMV are estimated and used to estimate the occluded hand joints HJ or hand-mesh vertices HMV. In some embodiments, a mesh-adjugate matrix502may be used to update the predicted attention mask PAM. For example, a first hand joint HJ (depicted by the number “1” inFIG.5E) may be obstructed, and a second hand joint HJ (depicted by the number “2” inFIG.5E) may be unobstructed. The mesh-adjugate matrix502may be used to remove the mask associated with the first hand joint HJ because the first hand joint is connected to the second hand joint. Accordingly, the dynamic mask-updating mechanism311may enable obstructed hand joints HJ or hand-mesh vertices to be more accurately estimated based on information associated with unobstructed hand joints HJ or hand-mesh vertices.

FIG.6is a block diagram of an electronic device in a network environment600, according to an embodiment.

Referring toFIG.6, an electronic device601in a network environment600may communicate with an electronic device602via a first network698(e.g., a short-range wireless communication network), or an electronic device604or a server608via a second network699(e.g., a long-range wireless communication network). The electronic device601may communicate with the electronic device604via the server608. The electronic device601may include a processor620, a memory630, an input device650, a sound output device655, a display device660, an audio module670, a sensor module676, an interface677, a haptic module679, a camera module680, a power management module688, a battery689, a communication module690, a subscriber identification module (SIM) card696, or an antenna module697. In one embodiment, at least one (e.g., the display device660or the camera module680) of the components may be omitted from the electronic device601, or one or more other components may be added to the electronic device601. Some of the components may be implemented as a single integrated circuit (IC). For example, the sensor module676(e.g., a fingerprint sensor, an iris sensor, or an illuminance sensor) may be embedded in the display device660(e.g., a display).

The processor620may execute software (e.g., a program640) to control at least one other component (e.g., a hardware or a software component) of the electronic device601coupled with the processor620and may perform various data processing or computations.

As at least part of the data processing or computations, the processor620may load a command or data received from another component (e.g., the sensor module676or the communication module690) in volatile memory632, process the command or the data stored in the volatile memory632, and store resulting data in non-volatile memory634. The processor620may include a main processor621(e.g., a CPU or an application processor (AP)), and an auxiliary processor623(e.g., a GPU, an image signal processor (ISP), a sensor hub processor, or a communication processor (CP)) that is operable independently from, or in conjunction with, the main processor621. Additionally or alternatively, the auxiliary processor623may be adapted to consume less power than the main processor621, or execute a particular function. The auxiliary processor623may be implemented as being separate from, or a part of, the main processor621.

The auxiliary processor623may control at least some of the functions or states related to at least one component (e.g., the display device660, the sensor module676, or the communication module690) among the components of the electronic device601, instead of the main processor621while the main processor621is in an inactive (e.g., sleep) state, or together with the main processor621while the main processor621is in an active state (e.g., executing an application). The auxiliary processor623(e.g., an image signal processor or a communication processor) may be implemented as part of another component (e.g., the camera module680or the communication module690) functionally related to the auxiliary processor623.

The memory630may store various data used by at least one component (e.g., the processor620or the sensor module676) of the electronic device601. The various data may include, for example, software (e.g., the program640) and input data or output data for a command related thereto. The memory630may include the volatile memory632or the non-volatile memory634.

The program640may be stored in the memory630as software, and may include, for example, an operating system (OS)642, middleware644, or an application646.

The input device650may receive a command or data to be used by another component (e.g., the processor620) of the electronic device601, from the outside (e.g., a user) of the electronic device601. The input device650may include, for example, a microphone, a mouse, or a keyboard.

The sound output device655may output sound signals to the outside of the electronic device601. The sound output device655may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as playing multimedia or recording, and the receiver may be used for receiving an incoming call. The receiver may be implemented as being separate from, or a part of, the speaker.

The display device660may visually provide information to the outside (e.g., a user) of the electronic device601. The display device660may include, for example, a display, a hologram device, or a projector and control circuitry to control a corresponding one of the display, hologram device, and projector. The display device660may include touch circuitry adapted to detect a touch, or sensor circuitry (e.g., a pressure sensor) adapted to measure the intensity of force incurred by the touch.

The audio module670may convert a sound into an electrical signal and vice versa. The audio module670may obtain the sound via the input device650or output the sound via the sound output device655or a headphone of an external electronic device602directly (e.g., wired) or wirelessly coupled with the electronic device601.

The sensor module676may detect an operational state (e.g., power or temperature) of the electronic device601or an environmental state (e.g., a state of a user) external to the electronic device601, and then generate an electrical signal or data value corresponding to the detected state. The sensor module676may include, for example, a gesture sensor, a gyro sensor, an atmospheric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

The interface677may support one or more specified protocols to be used for the electronic device601to be coupled with the external electronic device602directly (e.g., wired) or wirelessly. The interface677may include, for example, a high-definition multimedia interface (HDMI), a universal serial bus (USB) interface, a secure digital (SD) card interface, or an audio interface.

A connecting terminal678may include a connector via which the electronic device601may be physically connected with the external electronic device602. The connecting terminal678may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

The haptic module679may convert an electrical signal into a mechanical stimulus (e.g., a vibration or a movement) or an electrical stimulus which may be recognized by a user via tactile sensation or kinesthetic sensation. The haptic module679may include, for example, a motor, a piezoelectric element, or an electrical stimulator.

The camera module680may capture a still image or moving images. The camera module680may include one or more lenses, image sensors, image signal processors, or flashes. The power management module688may manage power supplied to the electronic device601. The power management module688may be implemented as at least part of, for example, a power management integrated circuit (PMIC).

The battery689may supply power to at least one component of the electronic device601. The battery689may include, for example, a primary cell which is not rechargeable, a secondary cell which is rechargeable, or a fuel cell.

The communication module690may support establishing a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device601and the external electronic device (e.g., the electronic device602, the electronic device604, or the server608) and performing communication via the established communication channel. The communication module690may include one or more communication processors that are operable independently from the processor620(e.g., the AP) and supports a direct (e.g., wired) communication or a wireless communication. The communication module690may include a wireless communication module692(e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module694(e.g., a local area network (LAN) communication module or a power line communication (PLC) module). A corresponding one of these communication modules may communicate with the external electronic device via the first network698(e.g., a short-range communication network, such as Bluetooth™, wireless-fidelity (Wi-Fi) direct, or a standard of the Infrared Data Association (IrDA)) or the second network699(e.g., a long-range communication network, such as a cellular network, the Internet, or a computer network (e.g., LAN or wide area network (WAN)). These various types of communication modules may be implemented as a single component (e.g., a single IC), or may be implemented as multiple components (e.g., multiple ICs) that are separate from each other. The wireless communication module692may identify and authenticate the electronic device601in a communication network, such as the first network698or the second network699, using subscriber information (e.g., international mobile subscriber identity (IMSI)) stored in the subscriber identification module696.

The antenna module697may transmit or receive a signal or power to or from the outside (e.g., the external electronic device) of the electronic device601. The antenna module697may include one or more antennas, and, therefrom, at least one antenna appropriate for a communication scheme used in the communication network, such as the first network698or the second network699, may be selected, for example, by the communication module690(e.g., the wireless communication module692). The signal or the power may then be transmitted or received between the communication module690and the external electronic device via the selected at least one antenna.

Commands or data may be transmitted or received between the electronic device601and the external electronic device604via the server608coupled with the second network699. Each of the electronic devices602and604may be a device of a same type as, or a different type from, the electronic device601. All or some of operations to be executed at the electronic device601may be executed at one or more of the external electronic devices602,604, or608. For example, if the electronic device601should perform a function or a service automatically, or in response to a request from a user or another device, the electronic device601, instead of, or in addition to, executing the function or the service, may request the one or more external electronic devices to perform at least part of the function or the service. The one or more external electronic devices receiving the request may perform the at least part of the function or the service requested, or an additional function or an additional service related to the request and transfer an outcome of the performing to the electronic device601. The electronic device601may provide the outcome, with or without further processing of the outcome, as at least part of a reply to the request. To that end, a cloud computing, distributed computing, or client-server computing technology may be used, for example.

FIG.7Ais a flowchart depicting example operations of a method of estimating an interaction with a device, according to some embodiments of the present disclosure.

Referring toFIG.7A, a method700A of estimating an interaction with a device100may include one or more of the following operations. An estimation model101may configure a first token T1and a second token T2of the estimation model101according to one or more first features of a 3D object (seeFIG.1andFIG.5C) (operation701A). The estimation model101may apply a first weight Wa to the first token T1to produce a first-weighted input token WIT1and may apply a second weight Wb that is different from the first weight Wa to the second token T2to produce a second-weighted input token WIT2(operation702A). A first encoder layer301of an estimation-model encoder120of the estimation model101may generate an output token OT based on the first-weighted input token WIT1and the second-weighted input token WIT2(operation703A).

FIG.7Bis a flowchart depicting example operations of a method of estimating an interaction with a device, according to some embodiments of the present disclosure.

Referring toFIG.7B, a method700B of estimating an interaction with a device100may include one or more of the following operations. A 2D feature-extraction model115may receive one or more first features corresponding to input data associated with an interaction with the device from a backbone110(operation701B). The 2D feature-extraction model115may extract one or more second features associated with the one or more first features, wherein the one or more second features include one or more 2D features (operation702B). The 2D feature-extraction model115may generate data based on the one or more 2D features (operation703B). The 2D feature-extraction model115may provide the data to an estimation-model encoder120of the estimation model101(operation704B).

FIG.7Cis a flowchart depicting example operations of a method of estimating an interaction with a device, according to some embodiments of the present disclosure.

Referring toFIG.7C, a method700C of estimating an interaction with a device100may include one or more of the following operations. A device100may generate a 3D scene including a visual representation of a 3D object (e.g., a body part) associated with an interaction with the device100(operation701C). The device100may update the visual representation of the 3D object based on an output token generated by an estimation model101associated with the device100(operation702C).

FIG.8is a flowchart depicting example operations of a method of estimating an interaction with a device, according to some embodiments of the present disclosure.

Referring toFIG.8, a method800of estimating an interaction with a device100may include one or more of the following operations. A backbone110of an estimation model101may receive input data12corresponding to an interaction with the device100(operation801). The backbone110may extract one or more first features from the input data12(operation802). The estimation model101may associate a first token T1and a second token T2of the estimation model101with the one or more first features (seeFIG.1andFIG.5C) (operation803). The estimation model101may apply a first weight Wa to the first token T1to produce a first-weighted input token WIT1and may apply a second weight Wb that is different from the first weight Wa to the second token T2to produce a second-weighted input token WIT2(operation804). A first encoder layer301of an estimation-model encoder120of the estimation model101may calculate an output token OT based on receiving the first-weighted input token WIT1and the second-weighted input token WIT2as inputs (operation805). A 2D feature-extraction model115may receive the one or more first features from the backbone110(operation806). The 2D feature-extraction model115may extract one or more second features associated with the one or more first features, wherein the one or more second features include one or more 2D features (operation807). The estimation-model encoder120may receive data generated based on the one or more 2D features (operation808). The estimation model may concatenate a token, associated with an output of a first BERT encoder, with camera intrinsic-parameter data or with 3D hand-wrist data to generate concatenated data, and receive the concatenated data at a second BERT encoder (operation809). The estimation model101may generate an estimated output32based on the output token OT and the data generated based on the one or more 2D features (operation810). The estimation model101may cause an operation to be performed on the device100based on the estimated output32(operation811).