Patent ID: 12254550

It will be noted that throughout the appended drawings, like features are identified by like reference numerals.

DETAILED DESCRIPTION

The description that follows describes example systems, methods, techniques, instruction sequences, and computing machine program products that comprise illustrative embodiments of the disclosure, individually or in combination. In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide an understanding of various embodiments of the inventive subject matter. It will be evident, however, to those skilled in the art, that various embodiments of the inventive subject matter may be practiced without these specific details.

The term ‘content’ used throughout the description herein should be understood to include all forms of media content items, including images, videos, audio, text, 3D models (e.g., including textures, materials, meshes, and more), animations, vector graphics, and the like.

The term ‘game’ used throughout the description herein should be understood to include video games and applications that execute and present video games on a device, and applications that execute and present simulations on a device. The term ‘game’ should also be understood to include programming code (either source code or executable binary code) which is used to create and execute the game on a device.

The term ‘environment’ used throughout the description herein should be understood to include 2D digital environments (e.g., 2D video game environments, 2D simulation environments, 2D content creation environments, and the like), 3D digital environments (e.g., 3D game environments, 3D simulation environments, 3D content creation environments, virtual reality environments, and the like), and augmented reality environments that include both a digital (e.g., virtual) component and a real-world component.

The term ‘digital object’, used throughout the description herein is understood to include any digital object or digital element within an environment. A digital object can represent (e.g., in a corresponding data structure) almost anything within the environment; including 3D models (e.g., characters, weapons, scene elements (e.g., buildings, trees, cars, treasures, and the like)) with 3D model textures, backgrounds (e.g., terrain, sky, and the like), lights, cameras, effects (e.g., sound and visual), animation, and more. The term ‘digital object’ may also be understood to include linked groups of individual digital objects. A digital object is associated with data that describes properties and behavior for the object.

The terms ‘asset’, ‘game asset’, and ‘digital asset’, used throughout the description herein are understood to include any data that can be used to describe a digital object or can be used to describe an aspect of a digital project (e.g., including: a game, a film, a software application). For example, an asset can include data for an image, a 3D model (textures, rigging, and the like), a group of 3D models (e.g., an entire scene), an audio sound, a video, animation, a 3D mesh and the like. The data describing an asset may be stored within a file, or may be contained within a collection of files, or may be compressed and stored in one file (e.g., a compressed file), or may be stored within a memory. The data describing an asset can be used to instantiate one or more digital objects within a game at runtime (e.g., during an execution of the game).

The term ‘build’ and ‘game build’ used throughout the description herein should be understood to include a compiled binary code of a game which can be executed on a device, and which, when executed can provide a playable version of the game (e.g., playable by a human or by an artificial intelligence agent).

The terms ‘client’ and ‘application client’ used throughout the description herein are understood to include a software client or software application that can access data and services on a server, including accessing over a network.

Throughout the description herein, the term “agent” and “AI agent” should be understood to include entities such as a non-player character (NPC), a robot, and a game world which are controlled by an artificial intelligence system or model.

Throughout the description herein, the term ‘mixed reality’ (MR) should be understood to include all combined environments in the spectrum between reality and virtual reality (VR) including virtual reality, augmented reality (AR) and augmented virtuality.

A method of generating or modifying poses in an animation of a character are disclosed. Variable numbers and types of supplied inputs are combined into a single input. The variable numbers and types of supplied inputs correspond to one or more effector constraints for one or more joints of the character. The single input is transformed into a pose embedding. The pose embedding includes a machine-learned representation of the single input. The pose embedding is expanded into a pose representation output. The pose representation output includes local rotation data and global position data for the one or more joints of the character.

The present disclosure includes apparatuses which perform one or more operations or one or more combinations of operations described herein, including data processing systems which perform these methods and computer readable media which when executed on data processing systems cause the systems to perform these methods, the operations or combinations of operations including non-routine and unconventional operations.

The systems and methods described herein include one or more components or operations that are non-routine or unconventional individually or when combined with one or more additional components or operations, for generating animation poses with deep machine learning using a machine learning (ML) pose prediction system in accordance with embodiments of the invention. In accordance with an embodiment, the ML pose prediction system100may be implemented within an application210or module within an application layer320as shown inFIG.3(details ofFIG.3are described below). The systems and methods described herein provide a number of valuable benefits to digital content creators since creating high-quality character animation is difficult and expensive. Games that exhibit high-quality animations are typically high-budget productions that can afford expert animators, usually having access to high-end animation software. The systems and methods described herein can provide the benefit of simplifying generation of high quality animations (e.g., in computer games) by allowing non-expert animators (e.g., and even non-animators) to create high quality animations. The systems and methods described herein can provide the benefit of a generic solution that can be applied to any type of character (e.g., human, dog, octopus, snake, etc.), and does not require manual intervention when switching between types. The systems and methods described herein, and particularly with the use of domain specific input data (as described below) can provide the benefit of producing natural-looking poses. In addition, the ML pose prediction system100sand methods described herein can be implemented as a component in a system for pose estimation and/or classification operating on a monocular video (e.g., in order to improve a quality of pose estimation and/or classification). In accordance with an embodiment, the ML pose prediction system100may be used as a component within a secondary system because the ML pose prediction system100uses an architecture that accepts (e.g., as input) a variable number and type of effector inputs and operates without requiring novel training (e.g., without requiring additional machine learning training due to a different input combination of effectors) for an arbitrary combination of effector inputs. The ML pose prediction system100architecture is described in detail below with respect toFIG.1,FIG.2A, andFIG.2B. For example, the ML pose prediction system100may be trained a first time using machine learning techniques, and require no additional machine learning training when presented with different combinations of effector inputs. Accordingly, it may be implemented within a postprocessing step of an additional system to improve realism and overall accuracy of pose estimation for the additional system.

In accordance with an embodiment, the systems and methods described herein provide an end-to-end solution for pose prediction: the ML pose prediction system100takes a variable number and type of input effectors and determines full body joint orientations without using an optimization process (e.g., after a machine learning prediction) that aligns all solid bodies and joints of a physical model.

Generating a Character Rig for the ML Pose Prediction System

In accordance with an embodiment, a rig is generated for a type of character, wherein the rig is a bone structure associated with a 3D model of the character, and wherein the rig is to be used by the ML pose prediction system100to pose the character. A type of character may be associated with a skeleton shape and configuration for the type (e.g., a bipedal human shaped animation skeleton for a human type character, a quadrupedal shaped animation skeleton for a dog type character, and the like). The systems and methods described herein can be applied to any type of character (e.g., to any shape or type of skeleton) including a bipedal human type, a quadrupedal type (e.g., dog, giraffe, elephant), other odd shaped types (e.g., octopus), and more. In accordance with an embodiment, a skeleton may include a hierarchical set of joints and may also include constraints on the joints (e.g., length of bones between joints, angular constraints, and more) which may provide a basic structure for the skeleton. For example, the systems and methods described herein do not use anything specifically limited to a single type of skeleton (e.g., the human body), nor do the systems and methods use any hard-coded constraints which might limit application. As such, the systems and methods described herein can be applied for posing various shaped skeletons such as a dog, or an octopus. In accordance with an embodiment, the generating of a rig may include generating an associated set of effectors for the rig, whereby each effector in the set can be used (e.g., by a machine learning system) to pose a part of a character via the rig. In accordance with an embodiment, effectors do not define a pose of a character, they provide constraints for a variable number of joints for which a final pose (e.g., at the output of the ML pose prediction system100) must satisfy. In accordance with an embodiment, there may be a small number of effectors defined as an input to the ML pose prediction system100(e.g., describing constraints for a small number of associated joints), and whereby the system100would determine a pose to satisfy the small number of effector constraints (e.g., the system100may find a representation (e.g., a pose embedding described below) for a pose that satisfies the effectors, and then generates a final character pose based on the pose embedding). In accordance with an embodiment, an effector of the set of effectors may be of a type, with the types of effectors including a positional effector, a rotational effector, and a look-at effector as described below:

Positional: In accordance with an embodiment, a positional effector includes data describing a position in a world space (e.g., world space coordinates). A positional effector can include subtypes:

Joint effector: In accordance with an embodiment, a joint effector may be a subtype of a positional effector that represents a position of a joint for a character (e.g., such as a desired position for a left foot of bipedal character). In accordance with an embodiment, a joint effector may be a restraint imposed on a joint of a character which forces the joint to occupy the position defined therein.

Reach effector: In accordance with an embodiment, a reach effector is a subtype of a positional effector that represents a desired target position in a world space (e.g., a target ‘future’ position for a joint effector). In accordance with an embodiment, a reach effector may be associated with a specific joint or joint effector, and may indicate a desired position for the joint. In accordance with an embodiment, a reach effector may not be associated with a specific joint or joint effector, but may indicate a desired position for a part of a character (e.g., a desired position for a left hand of a character to grab or point at).

look-at effector: In accordance with an embodiment, a look-at effector is an effector type that includes a 3D position which represents a desired target position in a world space for a joint, wherein the joint is forced (e.g., by the ML pose prediction system100) to orient itself towards the desired target position (e.g., the joint is forced to “look at” the target position). In accordance with an embodiment a look-effector provides an ability to maintain a global orientation of a joint towards a particular global position in a scene (for example, looking at a given object). The look-at effector is generic in that it allows a model of a neural network architecture102within the ML pose prediction system100(the neural network architecture102described below with respect toFIG.1) to align any direction within a joint (e.g., expressed in a local frame of reference), towards a global target location. In accordance with an embodiment, the look-at effector may include data describing the following: a 3D point (e.g., the desired target position), a joint (e.g., a specified joint within a character which must target the desired target position), and a specified axis of the joint which must orient itself to the 3D point (e.g., an axis of the joint which is forced by the ML pose prediction system to point at the 3D point, wherein the axis may be defined with any arbitrary unit-length vector defining an arbitrary local direction). In accordance with an embodiment, and during a training of the neural network architecture102, the network architecture102may be provided with a look-at effector (e.g., including a 3D point in an environment and a specified joint in a character), and may learn to generate a pose of the character wherein the specified joint will additionally satisfy a requirement to look at (e.g., point towards) the 3D point.

Rotational effector: In accordance with an embodiment, a rotational effector may include directional data (e.g., such as a direction vector or an amount and direction of rotation). For example, a directional effector may include a vector specifying a gaze direction, a running velocity, a hand orientation, and the like. In accordance with an embodiment, a rotational effector may include data which describes a local rotation or local direction which is described relative to an internal coordinate system of a character (e.g., a rotation relative to a character rig or relative to a set of joints for the character). In accordance with an embodiment, a rotational effector may include data which describes a global rotation or global direction which is described relative to a coordinate system which is external to the character (e.g., a rotation relative to a coordinate system external to a character rig or external to a set of joints for the character).

While positional, rotational, and look-at types are described above, embodiments of this present disclosure are not limited in this regard. Other effector types may be defined and used within the ML pose prediction system100without departing from the scope of this disclosure.

In accordance with an embodiment, an effector within the ML pose prediction system100includes associated embedded data which represents semantic information for the effector. A semantic meaning (e.g., encoded via an embedding) may be learned by machine learning techniques (e.g., including training and data augmentation as described herein) by the ML pose prediction system100(e.g., via a neural network therein, including the pose encoder140described below with respect toFIG.1), wherein the semantic meaning may include an intended use of an effector. The embedded data may enable online programmability of a neural network architecture (e.g., the neural network architecture102shown inFIG.1) within the ML pose prediction system100without requiring a retraining, wherein the online programmability refers to an ability to program the neural network for a new task without a requirement to retrain the neural network. For example, this may include an ability to process a first input that includes a first set of effectors with a first number and type, and to process a second input that includes a second set of effectors with a second number and type, wherein the processing of the first input and the second input are performed with the neural network without any retraining between the processing. The first set of effectors and the second set of effectors may be provided by an external input (e.g., an input of a user via a joystick, mouse, screen tap or other). For example, a user may specify a position for a hand, then provide a hip position, then provide a look-at position for a face, wherein the ML pose prediction system100can produce a new output pose based on the variable input over time. In accordance with an embodiment, the embedded data may be appended (e.g., within a vector data structure) to coordinate data, angle data, or other data associated with the effector.

In accordance with an embodiment, during a training of a neural network within the ML pose prediction system100(e.g., the neural network architecture102shown inFIG.1) and during an operation of a trained version of the neural network, an associated embedding for a joint effector may be used by the neural network within the ML pose prediction system100as an identifier (e.g., to determine which specific joint within a character is being processed).

In accordance with an embodiment, the embedded data associated with an effector includes data describing a type for the effector (e.g., wherein types may be described as above: positional, look-at, and directional). In accordance with an embodiment, the embedded type data may be appended to the effector data (e.g., within a vector data structure) so that during training and during operation (e.g., after a training), the neural network within the ML pose prediction system100(e.g., the neural network architecture102shown inFIG.1) is aware of a type of effector it is processing.

In accordance with an embodiment, the embedded data associated with an effector includes data describing a weight of the effector, wherein the weight describes a relative importance of the effector when compared to other effectors. In accordance with an embodiment, during training and during operation (e.g., after a training), a neural network within the ML pose prediction system100(e.g., the neural network architecture102shown inFIG.1) may use weight embedded data for an effector to determine a weighting of data associated with the effector (e.g., to determine a weighting of embedded data when using said data within the neural network described inFIG.1,FIG.2A, andFIG.2B). In accordance with an embodiment, as used during training and operation, the weight embedded data provides additional programmability to control a level of importance for each effector.

In accordance with an embodiment, the neural network within the ML pose prediction system100(e.g., the neural network architecture102shown inFIG.1) derives a set of parameters for one or more effectors using machine learning techniques. This may include determining how one or more effectors interact with a full body skeleton using machine learning techniques (e.g., during training). For example, this may include determining constraints (e.g., parameterization) using input data, such as twist or swing limits per joint, etc.

Architecture:

In accordance with an embodiment, and shown inFIG.1is a neural network architecture102for a pose prediction system100. The neural network architecture102includes an encoder (e.g., a pose encoder140) followed by a decoder (e.g., a pose decoder160) and generates an output prediction172from a set of inputs110. As shown inFIG.1, the neural network architecture102may generate the output172in a plurality of steps. In accordance with an embodiment, in a first step of the plurality of steps, a variable number and type of user supplied inputs may be processed, embedded (described below), and combined (e.g., concatenated130) into a single input136(e.g., a single input matrix) for the pose encoder140. The processing may include a processing for translation invariance122, padding126, concatenation128, and the like. The neural network architecture102is flexible in that it accepts a variable number and type of effector for each joint of a character. For example, any joint within an input character may have zero or more associated inputs, and an associated input may include one or more different types (e.g., a first joint may be constrained by user-specified 3D position coordinates and global rotation, while a second joint may be constrained with a look-at effector). In accordance with an embodiment, in a second step of the plurality of steps, the pose encoder140may transform the pose specified via effectors (e.g., the input136) into a single vector encoding of a pose (e.g., a pose embedding154). In accordance with an embodiment, in a third step of the plurality of steps, the posed decoder160may expand the pose embedding154into a full pose representation output172including local rotation data178and global position data176for each input joint.

In accordance with an embodiment, the translation invariance122may include a re-referencing of input positions relative to a centroid of input positional effectors to achieve translation invariance. The translation invariance122may simplify a handling of poses in global space while not relying on a precise reference frame, which can be difficult to define for heterogeneous MOCAP sources.

In accordance with an embodiment, the neural network architecture102does not require input to follow any specific scheme or that it be fully-specified. Instead, the neural network architecture102allows for complete flexibility of defining a character pose by accepting a variable number of inputs of different types. Accordingly, the neural network architecture102accepts any combination of input110that includes position effectors (3D coordinates), rotation effectors (with any 6DoF representation) and look-at effectors (3D coordinates). In accordance with an embodiment, and shown inFIG.1, an input effector may include data for position112and rotation114. In accordance with an embodiment, for mathematical convenience, input rotation data114may be in a 6 degree of freedom (6DoF) format that is described with six values. In accordance with an embodiment, input position data112(e.g., 3D position or look-at coordinates) may be padded (e.g., by adding 3 zero values at operation126) so that the input position data and the input rotation data114are the same length when provided to the pose encoder140. In accordance with an embodiment, each effector may be further characterized by tolerance data116, joint ID118, and type120. Tolerance may be a positive floating point value. A smaller tolerance value implies that an effector value has to be more strictly reproduced in a reconstructed output pose (e.g., within the output172). Joint ID for an effector may be a value (e.g., an integer) indicating which joint is affected by the effector. Effector type may be a value (e.g., an integer) indicating a positional, rotational, or look-at effector (e.g., for positional effector type=‘0’, for rotational effector type=‘1’, and for rotational effector type=‘2’). In accordance with an embodiment, categorical variables (type120and joint ID118) may be embedded into a continuous vector and may also be concatenated with the effector data (position112or rotation114), resulting in an input136to the pose encoder140being an matrix (e.g., a matrix with size N×Ein) with a number of rows (e.g., N rows as shown inFIG.1) corresponding to a number of input effectors and a number of columns (e.g., Eincolumns as shown inFIG.1) corresponding to a combined dimension (e.g., an embedding dimensionality) of all categorical variable embeddings plus 6 DoF effector input dimensions (e.g., either padded position data112or rotation data114). In accordance with an embodiment, as shown inFIG.1, a variable ‘B’ may denote a batch dimension for the input110.

In accordance with an embodiment, the pose encoder140may be a multi-stage residual neural network with residual links of forward and backward types interleaved with prototype layers (148,150, and152) of the forward links. In accordance with an embodiment, the pose encoder may applying a machine-learned model based on a fully-connected residual neural network architecture depicted inFIG.1(e.g., within the pose encoder140) andFIG.2B. In accordance with an embodiment, a prototype layer may be defined as a mean over the leading (effector) dimension of its input. Each stage of the pose encoder140corresponds to one residual block. The structure within a block is described below with respect toFIG.2B. The residual links may provide several benefits, including: (i) improving gradient flow and increasing network depth and (ii) achieving an interaction of encodings of individual joints with an encoding of an entire pose created at each encoder stage.

In accordance with an embodiment, as can be seen inFIG.1, a forward encoding of individual effectors is collapsed into a representation of a complete pose154via prototype layers. The representation of a complete pose154may be accumulated across a plurality of residual blocks (142,144,146) to form a final pose representation154as an output of the pose encoder140. In accordance with an embodiment, constant factors C1and C2may serve a purpose of aligning scales of a residual link from a block (e.g.,238fromFIG.2B) and a global prototype anion of a pose (e.g.,154).

Decoder

In accordance with embodiment, the pose decoder160may include two separate modules, both of which may be configured as a fully-connected residual (FCR) neural network architecture depicted inFIG.2AandFIG.2Bdescribed below. In accordance with an embodiment, a first module162of the two modules may be a global position decoder (GPD), wherein the GPD162predicts the internal pose representation154generated by the pose encoder140directly into an unconstrained prediction of 3D joint positions. The prediction by the global position decoder may be generated by applying a machine learned model based on a fully-connected residual neural network architecture depicted inFIG.2AandFIG.2B. In accordance with an embodiment, the output joint positions164may form a draft pose, in which bone constraints are not necessarily respected. In accordance with an embodiment, a second module168of the two modules may be an inverse kinematics decoder (IKD), wherein the inverse kinematics decoder168predicts internal geometric parameters (e.g., local rotation angles or joint rotations178) of the skeleton kinematic system. The prediction by the inverse kinematics decoder may be generated by applying a machine learned model based on a fully-connected residual neural network architecture depicted inFIG.2AandFIG.2B. In accordance with an embodiment, the inverse kinematic decoder168accepts a concatenation of (i) the pose embedding154generated by the pose encoder140and (ii) the (unconstrained) joint position predictions164generated by the global position decoder162. In accordance with an embodiment, the inverse kinematic decoder predicts the local rotations178of the skeleton joints that when subjected to predefined skeleton kinematic equations generate feasible coordinates of all joints.

Global Position Decoder: GPD

In accordance with an embodiment, based on the GPD162producing joint position predictions164without relying on skeleton constraints, the predictions may not respect skeleton topology and may not be physically feasible. The purpose of the GPD162module may be two-fold. First, the task of predicting unconstrained joint positions164may provide a task for generating a meaningful pose embedding. Second, the GPD module162may generate a reference point for the inverse kinematics decoder168.

In accordance with an embodiment, the inverse kinematics decoder module168generates local joint rotations178based on positions defined in global space. In order for the IKD168to provide correct rotations, an origin of the kinematic chain in world space must be provided to the IKD168, and the output of the GPD162provides this data.

Inverse Kinematics Decoder (IKD)

In accordance with an embodiment, the IKD168may accept a concatenation of (i) the pose embedding154generated by the pose encoder140and (ii) the predicted joint positions164(e.g., a pose draft) predicted by the GPD module162. In accordance with an embodiment, the IKD168may predict (e.g., using the concatenated input) the local rotation angles178of each joint. In accordance with an embodiment, the predicted local rotation angles178may also be processed via a forward kinematics pass170, which generates a global (e.g., and physically feasible) coordinates176of skeletal joints and global joint rotations. The forward kinematics pass is further described in more detail below.

Forward Kinematics Pass

In accordance with an embodiment, the forward kinematics pass170operates on the output178of the IKD168and translates the local joint rotations178and a global root position165into global joint rotations and global joint coordinates176. The global root position may be data describing a position of a joint defined as a root joint (e.g., within the input110) which may provide a reference point (e.g., an origin) for other joint positions within the input. In accordance with an embodiment, the global root position may be data describing a center of coordinates for the skeleton. In accordance with an embodiment, the translation operation of the forward kinematics pass170may be described by two matrices for each joint j, including an offset matrix and a rotation matrix, wherein the offset matrix of joint j provides displacements of the joint with respect to its parent joint along coordinates x, y, z when a rotation of joint j is zero. In accordance with an embodiment, the translation operation may use skeleton kinematic equations. In accordance with an embodiment, the offset matrix may be a fixed non-learnable matrix that describes bone length constraints for a skeleton. In accordance with an embodiment, the rotation matrix may be represented using Euler angles. However, in another embodiment, a more robust representation based on 6 element vectors predicted by the IKD module168may be used.

In accordance with an embodiment, the forward kinematics pass170takes the global root position165and rotation matrices of a plurality of joints as output by the IKD module168and generates a global rotation and global position of a joint of the plurality of joints by following a tree recursion from a parent joint of the joint.

In accordance with an embodiment, a global position and rotation matrix output for a joint (e.g., the output178of the forward kinematics pass170) may be a complete 6DOF prediction of the joint, including both global position and global rotation of the joint with respect to a center of coordinates for the skeleton.

In accordance with an embodiment, and shown inFIG.2is a schematic diagram of a neural network architecture202which may be used within the global position decoder162and the inverse kinematics decoder168. In accordance with an embodiment, the neural network architecture202has a fully connected residual neural network topology consisting of a plurality of fully connected blocks210connected using residual connections. In accordance with an embodiment, a block210may have a layer norm at the input and a fork at the output. A first output of the fork may produce a contribution to a global output220of the neural network architecture202. A second output of the fork may contribute to a residual connection a next block210, wherein the residual connection may additionally be processed by a non-linear rectifier function215(e.g., a ReLU non-linearity). As; shown inFIG.2A, there may be any number of layers consisting of a block210, an activation function215and a residual connection.

In accordance with an embodiment,FIG.2Bshows a neural network architecture within a block210. In accordance with an embodiment, an input230to the block210may pass through a plurality of fully connected layers232A,232L and more (collectively232). In accordance with an embodiment, an output from a final layer232L may pass through an activation function234to produce a block output240. The activation function234may be linear or non-linear. In accordance with an embodiment, a residual projection238may be created by combining an output from the final layer232L with the block input230and processing the combination with an activation function236. The activation function may be linear or non-linear (e.g., a ReLU activation function).

Losses within the Neural Network Architecture102

In accordance with an embodiment, three loss types may be used during a training of the neural network architecture102in a multi-task fashion. Individual loss terms may be combined additively (e.g., with loss weight factors for each) into a total loss term. The loss weight factors may be chosen to make sure that magnitudes of different loss terms have a same order of magnitude. A loss function combining rotation and position error terms via randomized weights based on randomly generated effector tolerance levels may be used.

In accordance with an embodiment, an L2 loss may be used as a loss type to penalize errors of 3D position predictions. The L2 loss may be defined as a mean squared error between a prediction and ground truth. In accordance with an embodiment, the L2 loss may be used to supervise output of the GPD module162(e.g., predicted joint positions164) by directly driving a learning process of GPD. In accordance with another embodiment, the L2 loss may be used to supervise the position output176of the forward kinematics pass170by indirectly driving a training of the IKD module168, wherein the IKD module168learns to produce local rotation angles that result in joint position predictions with small L2 loss after IKD outputs are subjected to the forward kinematics pass170.

In accordance with an embodiment, a geodesic loss may be used as a loss type to penalize errors in rotational output of the neural network architecture102. Geodesic loss may represent the smallest arc (in radians) to go from one rotation to another over a surface of a sphere. The geodesic loss may be defined for a ground truth rotation matrix and its prediction. The geodesic loss may be used to supervise the rotation output178of the IKD module168. The geodesic loss may directly drive a learning of the IKD module168by penalizing deviations with respect to a ground truth of local rotations of all joints.

In accordance with an embodiment, a combination of L2 loss and geodesic loss used when training the neural network architecture102may provide a benefit of allowing the neural network architecture102to learn a high-quality pose representation (e.g., as an output172). The combination of L2 loss and geodesic loss may be particularly beneficial for the neural network architecture102when reconstructing a partially specified pose, wherein multiple reconstructions may be plausible. Using the combination of L2 loss and geodesic loss may help to train the neural network architecture102to simultaneously reconstruct plausible joint positions and plausible joint rotations. In accordance with an embodiment, the combined training of the neural network architecture102on L2 loss and Geodesic loss may result in a synergistic effect, wherein the architecture102model trained on both L2 loss and geodesic loss generalizes better on both losses than a model trained only on one of the loss terms.

In accordance with an embodiment, a look-at loss may be used as a loss type, wherein the look-at loss is associated with look-at effector. In accordance with an embodiment, the look-at loss drives a learning of the IKD module168by penalizing deviations of global directions computed after the forward kinematics pass170with respect to a ground truth of global directions.

Training

In accordance with an embodiment, each stage of a ML pose prediction system100is a fully-connected neural network trained for a specific task as described above. In accordance with an embodiment, the training for the specific task includes performing data augmentation on input data, and designing training criterion to improve results of the ML pose prediction system100. In accordance with an embodiment, the training methodology described below includes a plurality of techniques to (i) regularize model training via data augmentation, (ii) teach the model to deal with incomplete and missing inputs and (iii) effectively combine loss terms for multi-task training. The data augmentation and the designing of training criterion is described below.

In accordance with an embodiment, a machine learning training process for the ML pose prediction system100requires as input a plurality of plausible poses for a type of character. In accordance with an embodiment, the plurality of plausible poses may be in the form of an animation clip (e.g., video clip). The input animation clips may be obtained from any existing animation clip repository (e.g., online video clips, proprietary animation clips, etc.), and may be generated specifically for the training (e.g., using motion capture).

In accordance with an embodiment, a ML pose prediction system100is trained for a type of character (e.g., requiring at least one ML pose prediction system100for posing per type of character). For example, there may be a ML pose prediction system100trained for human type characters, another ML pose prediction system100for dog type characters, another ML pose prediction system100for cat type characters, another ML pose prediction system100for snake type characters, and the like. The plurality of input poses to train an ML pose prediction system100can include any animation clips that include the type of character associated with the ML pose prediction system100. For example, an ML pose prediction system100for human posing would require that the ML pose prediction system100is trained using animation clips of human motion; whereas, an ML pose prediction system100for octopus posing would require that the ML pose prediction system100is trained using animation clips of octopus motion.

In accordance with an embodiment, a ML pose prediction system100is trained for a domain specific context that includes specific motions associated with the context, including boxing, climbing, sword fighting, and the like. A ML pose prediction system100may be trained for a specific domain context by using input animations for training of the ML pose prediction system100that includes animations specific to the domain context. For example, training a ML pose prediction system100for predicting fighting poses should include using a plurality of input fighting animation sequences.

Data Augmentation

In accordance with an embodiment, data augmentation may be used to artificially augment a size of an input training set (e.g., the plurality of input poses), the augmenting providing for an almost infinite motion data input. During training of an ML pose prediction system100, the data augmentation may include randomly translating and randomly rotating character poses in the plurality of input poses. The random translations may be performed in any direction. The addition of random translations of input poses may increase robustness of the ML pose prediction system100model by providing a greater range of input data. Furthermore, the addition of random translations can increase the possible applications of the ML pose prediction system100along with increasing the output quality of the ML pose prediction system100when posing a character. For example, the addition of random translations allows for the ML pose prediction system100to generate automatic body translation while generating a pose using a hierarchy of neural networks as described herein. For example, the ML pose prediction system100may generate a translation of a character in addition to providing a pose for the character in order to more closely match inputs (e.g., input effectors) to the generated output pose, since some generated poses may look more natural if accompanied by an additional translation. As a further example, consider a human character that includes input effectors describing position for the hands and feet, the addition of random translations during training will allow the ML pose prediction system100to predict a natural position of the character body in a world space from the input effectors of the hands and feet position. In accordance with an embodiment, the random rotations may only be performed around a vertical axis, as character poses are typically highly dependent on gravity. The addition of random rotation in input data is also important to train an ML pose prediction system100to learn automatic full or partial body rotation that may not be present in the original input data. Furthermore, the addition of random rotations also allows for the ML pose prediction system100to generate automatic body rotation while generating a pose using a hierarchy of neural networks as described herein. For example, the ML pose prediction system100may generate a rotation of a character in addition to providing a pose for the character in order to more closely match inputs (e.g., input effectors) to the generated output pose, since some generated poses may look more natural if accompanied by an additional rotation.

In accordance with an embodiment, the data augmentation may include augmentation based on selecting a plurality of different subsets of effectors as inputs (e.g. a first combination of hips and hands, a second combination could be head and feet, and the like). This leads to exponential growth in a number of unique training samples in a training dataset. The above described data augmentation, including a selecting of a plurality of different subsets of effectors as inputs, is possible with the network system because, as described here, the network system is configured to process semantic data of a variable number and type of input effectors. For example, the ML pose prediction system100model is not trained for a fixed number and type of inputs; instead, it is configured to handle any number of input effectors (and/or combinations of different effector types), each of which may have its own semantic meaning.

In accordance with an embodiment, the data augmentation may include augmentation based on a selecting of a plurality of different number of input effectors during training. For example, during training, the network may be forced to make predictions for all joints (e.g., for all joints in a character rig) based on any arbitrary subset of effector inputs. This can lead to a linear increase in a number of unique configurations of effectors. The above described data augmentation including a selecting of a plurality of different number of input effectors is possible with the network system because, as described here, the network system is configured to process semantic data of a variable number and type of input effectors.

In accordance with an embodiment, the data augmentation may include augmentation based on forcing a same encoder network to process random combinations of effector types during a training. Accordingly, a same encoder, with a same input may learn (e.g., during a training) to process both angular and positional measurements, increasing a flexibility of the trained network. For example, during a training, for any given sample, the network can be forced to predict all joints (e.g., for all joints in a character rig) based on a first combination of effector types (e.g., 3 joint positional effectors and 4 look-at effectors). In addition, for another sample, the network can be forced to predict all joints (e.g., for all joints in a character rig) based on a second combination of effector types (e.g., 10 joint positional effectors and 5 look-at effectors). The above described data augmentation including a processing of random combinations of effector types is possible with the network system because, as described here, the network system is configured to process semantic data of a variable number and type of input effectors.

In accordance with an embodiment, the data augmentation may include augmentation based on forcing a same encoder network to process input samples while randomly choosing a weight (e.g., importance level) for each effector. This results in an exponential growth of a number of unique input samples during training.

In accordance with an embodiment, the data augmentation may include augmentation based on adding random noise to coordinates and/or angles within each effector during a training. In accordance with an embodiment, a variance of the added noise during training may be configured so that it is synchronous with a weight (e.g., importance level) of an effector. This augmentation specifically forces the network to learn to respect certain effectors (e.g., effectors with a high weight) more than others (e.g., effectors with a low weight), on top of providing data augmentation. In accordance with an embodiment, data augmentation and training with the addition of random noise may have applications for processing results of monocular pose estimation, wherein each joint detection provided by a lower level pose estimation routine is accompanied with a measure of confidence.

In accordance with an embodiment, the data augmentation may be done on the fly during training to provide near infinite and variable input data for training (e.g., as opposed to pre-computing the data augmentation before training which only provides a fixed amount of input data). The on the fly data augmentation may also provide for a more variable input data set for training when compared to pre-computed data augmentation, by for example eliminating a possibility of using the same input data point (e.g., an input pose) twice since new input data is randomly generated when needed. For example, consider an original input data set of 1,000 poses, during a training, the ML pose prediction system100may generate additional input data via random translations and rotations as needed for training (e.g., based on a training metric). The generated additional input data during training may amount to 50,000 poses, 500,000 poses, 5 million poses or more and may be adjusted during training (e.g., depending on the training metric). This is in contrast to pre-computed data augmentation where data augmentation is computed before training and is fixed during training regardless of any training metric.

EXAMPLES

Using as few as 4 effectors for a whole human body (e.g., which may include 50 or more bones in a humanoid character rig) as inputs to a ML pose prediction system100(e.g., a ML pose prediction system100that uses the systems and methods described herein) allows for a prediction of realistic poses. For example, consider a ML pose prediction system100trained for predicting climbing poses of a human character with only the hands and feet used as inputs, and wherein an entire pose is predicted from the neural networks (e.g., without an external optimization process). In such a system, target positions for hands and feet can be linearly interpolated and body translation can be predicted by the ML pose prediction system100(rather than having body translation hardcoded or determined externally to the ML pose prediction system100).

In accordance with an embodiment, the systems and methods described herein can be applied for retargeting a specific animation. For example, a source animation involving boxing of a humanoid character may be used to control feet effectors as well as one hand effector of a character, while the remaining hand of the character is controlled by the ML pose prediction system100and moved toward a specific target, wherein the specific target is externally controlled and input to the ML pose prediction system100(e.g., by a user with a joystick). The 4 effectors are used to generate a fully dynamic punch animation.

In accordance with an embodiment, the ML pose prediction system100may be used to capture animation poses from a video clip that includes undetected joints (e.g., due to a wrong pose estimation or clipped image) while producing natural poses as an output. The above described of determining animation poses from a video clip that includes undetected joints is possible with the network system because, as described here, the network system is configured to process semantic data of a variable number and type of input effectors (e.g., such as an unpredictable nature of undetected joints).

While illustrated in the block diagrams as groups of discrete components communicating with each other via distinct data signal connections, it will be understood by those skilled in the art that the various embodiments may be provided by a combination of hardware and software components, with some components being implemented by a given function or operation of a hardware or software system, and many of the data paths illustrated being implemented by data communication within a computer application or operating system. The structure illustrated is thus provided for efficiency of teaching the present various embodiments.

It should be noted that the present disclosure can be carried out as a method, can be embodied in a system, a computer readable medium or an electrical or electro-magnetic signal. The embodiments described above and illustrated in the accompanying drawings are intended to be exemplary only. It will be evident to those skilled in the art that modifications may be made without departing from this disclosure. Such modifications are considered as possible variants and lie within the scope of the disclosure.

Certain embodiments are described herein as including logic or a number of components, modules, or mechanisms. Modules may constitute either software modules (e.g., code embodied on a machine-readable medium or in a transmission signal) or hardware modules. A “hardware module” is a tangible unit capable of performing certain operations and may be configured or arranged in a certain physical manner. In various example embodiments, one or more computer systems (e.g., a standalone computer system, a client computer system, or a server computer system) or one or more hardware modules of a computer system (e.g., a processor or a group of processors) may be configured by software (e.g., an application or application portion) as a hardware module that operates to perform certain operations as described herein.

In some embodiments, a hardware module may be implemented mechanically, electronically, or with any suitable combination thereof. For example, a hardware module may include dedicated circuitry or logic that is permanently configured to perform certain operations. For example, a hardware module may be a special-purpose processor, such as a field-programmable gate array (FPGA) or an Application Specific Integrated Circuit (ASIC). A hardware module may also include programmable logic or circuitry that is temporarily configured by software to perform certain operations. For example, a hardware module may include software encompassed within a general-purpose processor or other programmable processor. Such software may at least temporarily transform the general-purpose processor into a special-purpose processor. It will be appreciated that the decision to implement a hardware module mechanically, in dedicated and permanently configured circuitry, or in temporarily configured circuitry (e.g., configured by software) may be driven by cost and time considerations.

Accordingly, the phrase “hardware module” should be understood to encompass a tangible entity, be that an entity that is physically constructed, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) to operate in a certain manner or to perform certain operations described herein. As used herein, “hardware-implemented module” refers to a hardware module. Considering embodiments in which hardware modules are temporarily configured (e.g., programmed), each of the hardware modules need not be configured or instantiated at any one instance in time. For example, where a hardware module comprises a general-purpose processor configured by software to become a special-purpose processor, the general-purpose processor may be configured as respectively different special-purpose processors (e.g., comprising different hardware modules) at different times. Software may accordingly configure a particular processor or processors, for example, to constitute a particular hardware module at one instance of time and to constitute a different hardware module at a different instance of time.

Hardware modules can provide information to, and receive information from, other hardware modules. Accordingly, the described hardware modules may be regarded as being communicatively coupled. Where multiple hardware modules exist contemporaneously, communications may be achieved through signal transmission (e.g., over appropriate circuits and buses) between or among two or more of the hardware modules. In embodiments in which multiple hardware modules are configured or instantiated at different times, communications between such hardware modules may be achieved, for example, through the storage and retrieval of information in memory structures to which the multiple hardware modules have access. For example, one hardware module may perform an operation and store the output of that operation in a memory device to which it is communicatively coupled. A further hardware module may then, at a later time, access the memory device to retrieve and process the stored output. Hardware modules may also initiate communications with input or output devices, and can operate on a resource (e.g., a collection of information).

The various operations of example methods described herein may be performed, at least partially, by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such processors may constitute processor-implemented modules that operate to perform one or more operations or functions described herein. As used herein, “processor-implemented module” refers to a hardware module implemented using one or more processors.

Similarly, the methods described herein may be at least partially processor-implemented, with a particular processor or processors being an example of hardware. For example, at least some of the operations of a method may be performed by one or more processors or processor-implemented modules. Moreover, the one or more processors may also operate to support performance of the relevant operations in a “cloud computing” environment or as a “software as a service” (SaaS). For example, at least some of the operations may be performed by a group of computers (as examples of machines including processors), with these operations being accessible via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., an application program interface (API)).

The performance of certain of the operations may be distributed among the processors, not only residing within a single machine, but deployed across a number of machines. In some example embodiments, the processors or processor-implemented modules may be located in a single geographic location (e.g., within a home environment, an office environment, or a server farm). In other example embodiments, the processors or processor-implemented modules may be distributed across a number of geographic locations.

FIG.3is a block diagram300illustrating an example software architecture302, which may be used in conjunction with various hardware architectures herein described to provide a gaming engine3and/or components of the ML pose prediction system100.FIG.3is a non-limiting example of a software architecture and it will be appreciated that many other architectures may be implemented to facilitate the functionality described herein. The software architecture302may execute on hardware such as a machine400ofFIG.4that includes, among other things, processors410, memory430, and input/output (I/O) components450. A representative hardware layer304is illustrated and can represent, for example, the machine400ofFIG.4. The representative hardware layer304includes a processing unit306having associated executable instructions308. The executable instructions308represent the executable instructions of the software architecture302, including implementation of the methods, modules and so forth described herein. The hardware layer304also includes memory/storage310, which also includes the executable instructions308. The hardware layer304may also comprise other hardware312.

In the example architecture ofFIG.3, the software architecture302may be conceptualized as a stack of layers where each layer provides particular functionality. For example, the software architecture302may include layers such as an operating system314, libraries316, frameworks or middleware318, applications320and a presentation layer344. Operationally, the applications320and/or other components within the layers may invoke application programming interface (API) calls324through the software stack and receive a response as messages326. The layers illustrated are representative in nature and not all software architectures have all layers. For example, some mobile or special purpose operating systems may not provide the frameworks/middleware318, while others may provide such a layer. Other software architectures may include additional or different layers.

The operating system314may manage hardware resources and provide common services. The operating system314may include, for example, a kernel328, services330, and drivers332. The kernel328may act as an abstraction layer between the hardware and the other software layers. For example, the kernel328may be responsible for memory management, processor management (e.g., scheduling), component management, networking, security settings, and so on. The services330may provide other common services for the other software layers. The drivers332may be responsible for controlling or interfacing with the underlying hardware. For instance, the drivers332may include display drivers, camera drivers, Bluetooth® drivers, flash memory drivers, serial communication drivers (e.g., Universal Serial Bus (USB) drivers), Wi-Fi® drivers, audio drivers, power management drivers, and so forth depending on the hardware configuration.

The libraries316may provide a common infrastructure that may be used by the applications320and/or other components and/or layers. The libraries316typically provide functionality that allows other software modules to perform tasks in an easier fashion than to interface directly with the underlying operating system314functionality (e.g., kernel328, services330and/or drivers332). The libraries416may include system libraries334(e.g., C standard library) that may provide functions such as memory allocation functions, string manipulation functions, mathematic functions, and the like. In addition, the libraries316may include API libraries336such as media libraries (e.g., libraries to support presentation and manipulation of various media format such as MPEG4, H.264, MP3, AAC, AMR, JPG, PNG), graphics libraries (e.g., an OpenGL framework that may be used to render 2D and 3D graphic content on a display), database libraries (e.g., SQLite that may provide various relational database functions), web libraries (e.g., WebKit that may provide web browsing functionality), and the like. The libraries316may also include a wide variety of other libraries338to provide many other APIs to the applications320and other software components/modules.

The frameworks318(also sometimes referred to as middleware) provide a higher-level common infrastructure that may be used by the applications320and/or other software components/modules. For example, the frameworks/middleware318may provide various graphic user interface (GUI) functions, high-level resource management, high-level location services, and so forth. The frameworks/middleware318may provide a broad spectrum of other APIs that may be utilized by the applications320and/or other software components/modules, some of which may be specific to a particular operating system or platform.

The applications320include built-in applications340and/or third-party applications342. Examples of representative built-in applications340may include, but are not limited to, a contacts application, a browser application, a book reader application, a location application, a media application, a messaging application, and/or a game application. Third-party applications342may include any an application developed using the Android™ or iOS™ software development kit (SDK) by an entity other than the vendor of the particular platform, and may be mobile software running on a mobile operating system such as iOS™, Android™, Windows® Phone, or other mobile operating systems. The third-party applications342may invoke the API calls324provided by the mobile operating system such as operating system314to facilitate functionality described herein. In accordance with an embodiment, the applications320may include a ML pose prediction module370which may implement the ML pose prediction system100.

The applications320may use built-in operating system functions (e.g., kernel328, services330and/or drivers332), libraries316, or frameworks/middleware318to create user interfaces to interact with users of the system. Alternatively, or additionally, in some systems, interactions with a user may occur through a presentation layer, such as the presentation layer344. In these systems, the application/module “logic” can be separated from the aspects of the application/module that interact with a user.

Some software architectures use virtual machines. In the example ofFIG.3, this is illustrated by a virtual machine348. The virtual machine348creates a software environment where applications/modules can execute as if they were executing on a hardware machine (such as the machine400ofFIG.4, for example). The virtual machine348is hosted by a host operating system (e.g., operating system314) and typically, although not always, has a virtual machine monitor346, which manages the operation of the virtual machine348as well as the interface with the host operating system (i.e., operating system314). A software architecture executes within the virtual machine348such as an operating system (OS)350, libraries352, frameworks354, applications356, and/or a presentation layer358. These layers of software architecture executing within the virtual machine348can be the same as corresponding layers previously described or may be different.

FIG.4is a block diagram illustrating components of a machine400, according to some example embodiments, configured to read instructions from a machine-readable medium (e.g., a machine-readable storage medium) and perform any one or more of the methodologies discussed herein. Specifically,FIG.4shows a diagrammatic representation of the machine400in the example form of a computer system, within which instructions416(e.g., software, a program, an application, an applet, an app, or other executable code) for causing the machine400to perform any one or more of the methodologies discussed herein may be executed. As such, the instructions416may be used to implement modules or components described herein. The instructions transform the general, non-programmed machine into a particular machine programmed to carry out the described and illustrated functions in the manner described. In alternative embodiments, the machine400operates as a standalone device or may be coupled (e.g., networked) to other machines. In a networked deployment, the machine400may operate in the capacity of a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine400may comprise, but not be limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system, a cellular telephone, a smart phone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a web appliance, a network router, a network switch, a network bridge, or any machine capable of executing the instructions416, sequentially or otherwise, that specify actions to be taken by the machine400. Further, while only a single machine400is illustrated, the term “machine” shall also be taken to include a collection of machines that individually or jointly execute the instructions416to perform any one or more of the methodologies discussed herein.

The machine400may include processors410, memory430, and input/output (I/O) components450, which may be configured to communicate with each other such as via a bus402. In an example embodiment, the processors410(e.g., a Central Processing Unit (CPU), a Reduced Instruction Set Computing (RISC) processor, a Complex Instruction Set Computing (CISC) processor, a Graphics Processing Unit (GPU), a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Radio-Frequency Integrated Circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, a processor412and a processor414that may execute the instructions416. The term “processor” is intended to include multi-core processor that may comprise two or more independent processors (sometimes referred to as “cores”) that may execute instructions contemporaneously. AlthoughFIG.4shows multiple processors, the machine400may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiples cores, or any combination thereof.

The memory/storage430may include a memory, such as a main memory432, a static memory434, or other memory, and a storage unit436, both accessible to the processors410such as via the bus402. The storage unit436and memory432,434store the instructions416embodying any one or more of the methodologies or functions described herein. The instructions416may also reside, completely or partially, within the memory432,434, within the storage unit436, within at least one of the processors410(e.g., within the processor's cache memory), or any suitable combination thereof, during execution thereof by the machine400. Accordingly, the memory432,434, the storage unit436, and the memory of processors410are examples of machine-readable media438.

As used herein, “machine-readable medium” means a device able to store instructions and data temporarily or permanently and may include, but is not limited to, random-access memory (RAM), read-only memory (ROM), buffer memory, flash memory, optical media, magnetic media, cache memory, other types of storage (e.g., Erasable Programmable Read-Only Memory (EEPROM)) and/or any suitable combination thereof. The term “machine-readable medium” should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, or associated caches and servers) able to store the instructions416. The term “machine-readable medium” shall also be taken to include any medium, or combination of multiple media, that is capable of storing instructions (e.g., instructions416) for execution by a machine (e.g., machine400), such that the instructions, when executed by one or more processors of the machine400(e.g., processors410), cause the machine400to perform any one or more of the methodologies or operations, including non-routine or unconventional methodologies or operations, or non-routine or unconventional combinations of methodologies or operations, described herein. Accordingly, a “machine-readable medium” refers to a single storage apparatus or device, as well as “cloud-based” storage systems or storage networks that include multiple storage apparatus or devices. The term “machine-readable medium” excludes signals per se.

The input/output (I/O) components450may include a wide variety of components to receive input, provide output, produce output, transmit information, exchange information, capture measurements, and so on. The specific input/output (I/O) components450that are included in a particular machine will depend on the type of machine. For example, portable machines such as mobile phones will likely include a touch input device or other such input mechanisms, while a headless server machine will likely not include such a touch input device. It will be appreciated that the input/output (I/O) components450may include many other components that are not shown inFIG.4. The input/output (I/O) components450are grouped according to functionality merely for simplifying the following discussion and the grouping is in no way limiting. In various example embodiments, the input/output (I/O) components450may include output components452and input components454. The output components452may include visual components (e.g., a display such as a plasma display panel (PDP), a light emitting diode (LED) display, a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)), acoustic components (e.g., speakers), haptic components (e.g., a vibratory motor, resistance mechanisms), other signal generators, and so forth. The input components454may include alphanumeric input components (e.g., a keyboard, a touch screen configured to receive alphanumeric input, a photo-optical keyboard, or other alphanumeric input components), point based input components (e.g., a mouse, a touchpad, a trackball, a joystick, a motion sensor, or another pointing instrument), tactile input components (e.g., a physical button, a touch screen that provides location and/or force of touches or touch gestures, or other tactile input components), audio input components (e.g., a microphone), and the like.

In further example embodiments, the input/output (I/O) components450may include biometric components456, motion components458, environmental components460, or position components462, among a wide array of other components. For example, the biometric components456may include components to detect expressions (e.g., hand expressions, facial expressions, vocal expressions, body gestures, or eye tracking), measure biosignals (e.g., blood pressure, heart rate, body temperature, perspiration, or brain waves), identify a person (e.g., voice identification, retinal identification, facial identification, fingerprint identification, or electroencephalogram based identification), and the like. The motion components458may include acceleration sensor components (e.g., accelerometer), gravitation sensor components, rotation sensor components (e.g., gyroscope), and so forth. The environmental components460may include, for example, illumination sensor components (e.g., photometer), temperature sensor components (e.g., one or more thermometers that detect ambient temperature), humidity sensor components, pressure sensor components (e.g., barometer), acoustic sensor components (e.g., one or more microphones that detect background noise), proximity sensor components (e.g., infrared sensors that detect nearby objects), gas sensors (e.g., gas detection sensors to detection concentrations of hazardous gases for safety or to measure pollutants in the atmosphere), or other components that may provide indications, measurements, or signals corresponding to a surrounding physical environment. The position components462may include location sensor components (e.g., a Global Position System (GPS) receiver component), altitude sensor components (e.g., altimeters or barometers that detect air pressure from which altitude may be derived), orientation sensor components (e.g., magnetometers), and the like.

Communication may be implemented using a wide variety of technologies. The input/output (I/O) components450may include communication components464operable to couple the machine400to a network480or devices470via a coupling482and a coupling472respectively. For example, the communication components464may include a network interface component or other suitable device to interface with the network480. In further examples, the communication components464may include wired communication components, wireless communication components, cellular communication components, Near Field Communication (NFC) components, Bluetooth® components (e.g., Bluetooth® Low Energy), Wi-Fi® components, and other communication components to provide communication via other modalities. The devices470may be another machine or any of a wide variety of peripheral devices (e.g., a peripheral device coupled via a Universal Serial Bus (USB)).

Moreover, the communication components464may detect identifiers or include components operable to detect identifiers. For example, the communication components464may include Radio Frequency Identification (RFID) tag reader components, NFC smart tag detection components, optical reader components (e.g., an optical sensor to detect one-dimensional bar codes such as Universal Product Code (UPC) bar code, multi-dimensional bar codes such as Quick Response (QR) code, Aztec code, Data Matrix, Dataglyph, MaxiCode, PDF417, Ultra Code, UCC RSS-2D bar code, and other optical codes), or acoustic detection components (e.g., microphones to identify tagged audio signals). In addition, a variety of information may be derived via the communication components462, such as, location via Internet Protocol (IP) geo-location, location via Wi-Fi® signal triangulation, location via detecting a NFC beacon signal that may indicate a particular location, and so forth.

Throughout this specification, plural instances may implement components, operations, or structures described as a single instance. Although individual operations of one or more methods are illustrated and described as separate operations, one or more of the individual operations may be performed concurrently, and nothing requires that the operations be performed in the order illustrated. Structures and functionality presented as separate components in example configurations may be implemented as a combined structure or component. Similarly, structures and functionality presented as a single component may be implemented as separate components. These and other variations, modifications, additions, and improvements fall within the scope of the subject matter herein.

The embodiments illustrated herein are described in sufficient detail to enable those skilled in the art to practice the teachings disclosed. Other embodiments may be used and derived therefrom, such that structural and logical substitutions and changes may be made without departing from the scope of this disclosure. The Detailed Description, therefore, is not to be taken in a limiting sense, and the scope of various embodiments is defined only by the appended claims, along with the full range of equivalents to which such claims are entitled.

As used herein, the term “or” may be construed in either an inclusive or exclusive sense. Moreover, plural instances may be provided for resources, operations, or structures described herein as a single instance. Additionally, boundaries between various resources, operations, modules, engines, and data stores are somewhat arbitrary, and particular operations are illustrated in a context of specific illustrative configurations. Other allocations of functionality are envisioned and may fall within a scope of various embodiments of the present disclosure. In general, structures and functionality presented as separate resources in the example configurations may be implemented as a combined structure or resource. Similarly, structures and functionality presented as a single resource may be implemented as separate resources. These and other variations, modifications, additions, and improvements fall within the scope of embodiments of the present disclosure as represented by the appended claims. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense.