Patent ID: 12216815

DESCRIPTION OF EXAMPLE EMBODIMENTS

FIG.1illustrates an example artificial reality system100. In particular embodiments, the artificial reality system100may comprise a headset104, a controller106, and a computing system108. A user102may wear the headset104that may display visual artificial reality content to the user102. The headset104may include an audio device that may provide audio artificial reality content to the user102. The headset104may comprise one or more cameras110which can capture images and videos of environments. The headset104may include an eye tracking system to determine a vergence distance of the user102. A vergence distance may be a distance from the user's eyes to objects (e.g., real-world objects or virtual objects in a virtual space) that the user's eyes are converged at. The headset104may be referred to as a head-mounted display (HMD). In particular embodiments computing system108may determine a pose of headset104associated with user102. The headset pose may be determined by utilizing any of the sensor data or image data received by the computing system108.

One or more controllers106may be paired with the artificial reality system100. In particular embodiments one or more controllers106may be equipped with at least one inertial measurement units (IMUs) and infrared (IR) light emitting diodes (LEDs) for the artificial reality system100to estimate a pose of the controller and/or to track a location of the controller, such that the user may perform certain functions via the controller. The one or more controllers106may be equipped with one or more trackable markers distributed to be tracked by the computing system108. The one or more controllers106may comprise a trackpad and one or more buttons. The one or more controllers106may receive inputs from the user102and relay the inputs to the computing system108. The one or more controllers106may also provide haptic feedback to the user102. The computing system108may be connected to the headset104and the one or more controllers106through cables or wireless connections. The one or more controllers106may include a combination of hardware, software, and/or firmware not explicitly shown herein so as not to obscure other aspects of the disclosure.

In particular embodiments the computing system108may receive sensor data from one or more sensors of artificial reality system100. In particular embodiments the one or more sensors may be coupled to user102. In particular embodiments, the one or more sensors may be associated with the headset104worn by the user. For example and not by way of limitation, the headset104may include a gyroscope or inertial measurement unit that tracks the user's real-time movements and output sensor data to represent or describe the movement. The sensor data provided by such motion-tracking sensors may be used by the VR application to determine the user's current orientation and provide that orientation to the rendering engine to orient/reorient the virtual camera in the 3D space. As another example and not by way of limitation, the one or more controllers106may include inertial measurement units (IMUs) and infrared (IR) light emitting diodes (LEDs) configured to collect and send IMU sensor data to the computing system108. In particular embodiments the computing system108may utilize one or more sensor data with one or more tracking techniques, for example and not by way of limitation, SLAM tracking or IR-based tracking, to determine a pose of one or more components of artificial reality system100.

In particular embodiments the computing system108may receive one or more image data from one or more components of artificial reality system100. In particular embodiments this image data comprises image data captured from one or more cameras110associated with artificial reality system100. For example,FIG.1depicts one or more cameras110coupled within headset104. These one or more cameras may be positioned to capture one or more images associated with various perspectives, for example and not by way of limitation, one or more cameras associated with headset104that face downward (e.g. towards the feet of user102while standing).

In particular embodiments computing system108may determine a controller pose of one or more controllers106associated with user102. The controller pose associated with user102may be determined by utilizing sensor data or image data received by the computing system108, and utilizing one or more techniques, for example and not by way of limitation, computer vision techniques (e.g., image classification). Methods for determining controller poses are described further in U.S. application Ser. No. 16/734,172, filed Jan. 3, 2020, entitled “Joint Infrared and Visible Light Visual-Inertial Object Tracking,” hereby incorporated by reference in its entirety.

In particular embodiments the computing system108may control the headset104and the one or more controllers106to provide the artificial reality content to and receive inputs from the user102. The computing system108may be a standalone host computer system, an on-board computer system integrated with the headset104, a mobile device, or any other hardware platform capable of providing artificial reality content to and receiving inputs from the user102.

FIG.2illustrates a sample full body pose associated with user102. In particular embodiments, computing system108may generate a full body pose200associated with user102, comprising an upper body pose205and a lower body pose215. Full body pose200associated with user102may attempt to replicate a position and orientation of one or more joints of a user102utilizing artificial reality system100at a particular time. In particular embodiments, full body pose200associated with user102comprises a skeletal frame of inverse kinematics (“skeleton” or “body pose”), which may comprise a list of one or more joints.

Upper body pose205may correspond to a portion of the body of the user102. In particular embodiments upper body pose205may correspond to a portion of the body of user102that comprises at least a head and arm of the user102. In particular embodiments upper body pose205may be generated by determining a plurality of poses corresponding to a plurality of predetermined body parts or joints of the user102, for example, the head or wrist of the user102. In particular embodiments the upper body pose205may further comprise, for example and not by way of limitation, a pose of one or more joints associated with the upper body of user102, for example and not by way of limitation, a head pose210, a wrist pose220, an elbow pose220, a shoulder pose240, a neck pose250, or an upper spine pose260.

Lower body pose215may correspond to a portion of the body of user102. In particular embodiments lower body pose215may correspond to a portion of the body of user102that comprises at least a leg of the user102. In particular embodiments the lower body pose215may further comprise, for example and not by way of limitation, a joint pose of one or more joints associated with the lower body of user102, for example and not by way of limitation, a lower spine pose270, a hip pose280, a knee pose290, or an ankle pose295.

In particular embodiments the one or more joints poses comprising the full body pose200, the upper body pose205, or the lower body pose215may be represented through, for example and not by way of limitation, a subset of parameters that represent a position and/or orientation of each joint in a body pose200. The non-linear solver may parametrize each joint pose associated with user102by 7 degrees of freedom: 3 translation values (e.g., x, y, z), 3 rotation values (e.g., Euler angles in radians), and 1 uniform scale value. In particular embodiments these parameters may be represented using one or more coordinate systems, for example and not by way of limitation, via an absolute global coordinate system (e.g., x, y, z) or via a localized coordinate system relative to a parent joint, for example and not by way of limitation, a head joint.

In particular embodiments full body pose200, upper body pose205, or lower body pose215, and one or more joint poses comprising the full body pose200, upper body pose205, or lower body pose215, may be determined using received sensor or image data from one or more components of artificial reality system100. In particular embodiments one or more joint poses comprising full body pose200may be generated using a combination of one or more techniques, for example and not by way of limitation localization techniques (e.g., SLAM), machine learning techniques (e.g., a neural network), known spatial relationships with one or more artificial reality system components (e.g., a known spatial relationship between headset104and head pose210), visualization techniques (e.g., image segmentation), or optimization techniques (e.g., a non-linear solver). In particular embodiments one or more of these techniques may be utilized separately or in conjunction with one or more other techniques. This full body pose200associated with user102may be useful for a variety of applications as described herein.

In particular embodiments, the computing system108may generate an upper body pose205of the user102. The upper body pose205may be generated based on at least sensor data or image data. In particular embodiments, computing system108may determine upper body pose200utilizing one or more poses corresponding to a plurality of predetermined body parts of user102, for example and not by way of limitation, a joint pose of one or more joints comprising upper body pose205(e.g., a head pose210or a wrist pose220).

In particular embodiments the one or more poses may comprise a head pose210associated with the user102. The head pose210may comprise a location and an orientation of the head joint of user102while wearing the headset104. The head pose210associated with user102may be determined by computing system108utilizing any of the sensor data and/or image data received by the computing system108. In particular embodiments this head pose210associated with user102may be determined based on the pose of the headset104and a known spatial relationship between the headset104and the head of user102. In particular embodiments the head pose210associated with user102may be determined based on sensor data associated with headset104worn by user102.

In particular embodiments the one or more poses may comprise a wrist pose220associated with the user102. The wrist pose220may comprise a location and an orientation of a wrist joint of user102while interacting with artificial reality system100. The wrist pose220associated with user102may be determined by computing system108utilizing any of the sensor data and/or image data received by the computing system108. In particular embodiments the image data may comprise one or more images captured by the headset104worn by the user102. These one or more images may depict a wrist of the user102, or a device held by user102, (e.g., a controller106). In particular embodiments the computing system108may use one or more computer vision techniques, for example and not by way of limitation, image classification or object detection, to determine the wrist pose220utilizing the one or more images. In particular embodiments the wrist pose220may be determined based on the controller pose and a known spatial relationship between the controller106and the wrist of user102.

In particular embodiments, upper body pose205may be inferred based on the plurality of poses, for example and not by way of limitation, a head pose210or a wrist pose220. In particular embodiments the upper body pose205may be inferred by a non-linear kinematic optimization solver (“non-linear solver”) to infer one or more joint poses that comprise upper body pose205. In particular embodiments the non-linear solver may comprise a C++ library built for inverse kinematics with a large set of common error functions that cover a wide range of applications. In particular embodiments the non-linear solver may provide one or more helper functions for tasks that are usually related to global inverse kinematics problems (e.g., joint and skeleton structures, meshes and linear-blend skinning, error functions for common constraints) or one or more helper functions for mesh deformations (e.g., Laplacian surface deformation) or one or more IO functions for various file formats.

In particular embodiments the non-linear solver may infer one or more joint poses comprising upper body pose205. The one or more joint poses comprising upper body pose205associated with user102inferred by the non-linear solver may comprise a kinematic hierarchy at a particular time or state which is stored as a list of one or more joint poses. In particular embodiments the non-linear solver may include one or more basic solvers supported to infer the pose of one or more joints comprising upper body pose205, for example and not by way of limitation a L-BFGS or a Gauss-Newton solver. In particular embodiments the non-linear solver may utilize a skeletal solver function to solve for a single frame of inverse kinematics (a single body pose). For example, the skeletal solver function may take a current one or more parameters (e.g., a joint pose) set as an input and optimize the activated subset of parameters given the defined error functions. The convention of the non-linear solver is to minimize the error value of the current function (e.g., the skeletal solver function) to infer an accurate upper body pose205.

In particular embodiments the non-linear solver may represent the one or more joints poses through, for example and not by way of limitation, a subset of parameters that represent a position or orientation of each joint in an upper body pose205. In particular embodiments the non-linear solver may parametrize each joint pose by 7 degrees of freedom: 3 translation values (e.g., x, y, z), 3 rotation values (e.g., Euler angles in radians), and 1 uniform scale value. In particular embodiments these parameters may be represented using one or more coordinate systems, for example and not by way of limitation, via an absolute global coordinate system (e.g., x, y, z) or via a localized coordinate system relative to a parent joint, for example and not by way of limitation, a head joint.

In particular embodiments the non-linear solver may assign one or more variable limits for each joint pose parameter (e.g., a minimum or maximum value). In particular embodiments the non-linear solver may assign predetermined static weights to each joint pose parameter. These predetermined static weights may be determined, for example and not by way of limitation, based on the accuracy of the sensor data used to determine the value of each variable. For example, the joint pose parameters representing the head pose210and wrist pose220associated with user102may be assigned a higher static weight because they are determined with more accurate methods, such as SLAM techniques as described herein, than one or more other joint poses or variables within a joint pose parameter. In particular embodiments, the non-linear solver may use the joint parameters and predetermined static weights to infer an upper body pose205, which infers the most likely poses of one or more joints of user102at a particular time or state.

While these techniques, utilized either alone or in combination, typically allow for reliable determination of the upper body pose205and associated joints that comprise upper body pose2015(e.g., the head pose210, wrist pose220, elbow pose230, etc.), these techniques may be unreliable and inaccurate for determining lower body pose215and the associated joints that comprise lower body pose215(e.g., a hip pose280, a knee pose290, an ankle pose295, etc.). Due to user102only wearing headset104and one or more controllers106, limitations may exist in the sensor or image data associated with artificial reality system100. For example, image data captured from one or more cameras110may not contain at least a portion of the lower body of user102, resulting in little information with which to determine or generate an accurate lower body pose215associated with user102.

To remedy this problem, in particular embodiments computing system108may utilize one or more techniques described herein to generate a lower body pose215of the user102. In particular embodiments the lower body pose215may be generated by processing the upper body pose205using a machine-learning model.FIG.3illustrates an example of generating a lower body pose utilizing an inputted upper body pose. In particular,FIG.3illustrates an inputted upper body pose205associated with a user102of artificial reality system100generated using the methods described herein and received by machine-learning model300. In particular embodiments, the machine-learning model300may be based on a Generative Adversarial Network (GAN). Using particular embodiments described herein, machine-learning model300may utilize upper body pose205to generate a lower body pose215. In particular embodiments, the computing system may combine the generated upper body pose205with the generated lower body pose215to generate a full body pose in lieu of or in addition to the lower body pose215generated by the machine-learning model.

FIG.4illustrates a configuration for training a Generative Adversarial Network (GAN)400for pose prediction. GAN may include two separate neural networks, a Generator405(interchangeably referred to as “G” herein) and a Discriminator410(interchangeably referred to as “D” herein). In particular embodiments, the Generator405and the Discriminator410may be implemented as, for example and not by way of limitation, a neural network, although any network architecture suitable for the operations described herein may be utilized. At a high level, the Generator405may be configured to receive as input a generated upper body pose205and output a generated lower body pose215. In particular embodiments the upper body pose205may be combined with the lower body pose215to generate a full body pose425. In particular embodiments the Discriminator410may be configured to discriminate between “fake” full body poses425(those comprising lower body poses outputted by the Generator405) and “real” training lower body poses435from a training pose database440that are not generated by Generator405. In particular embodiments, the one or more training full body poses435may comprise full body poses from one or more images. The Generator405and the Discriminator410may be considered as adversaries, because the objective of the Generator405is to generate fake poses that would fool the Discriminator410(in other words, to increase the Discriminator's410error rate), and the objective of the Discriminator410is to correctly distinguish “fake” poses from the Generator405and “real” poses. In particular embodiments, the machine-learning model (e.g., the Generator405) is optimized during training to cause a second machine-learning model (e.g., the Discriminator410) to incorrectly determine that a given full body pose, generated using the machine-learning model, is unlikely generated using the machine-learning model.

In particular embodiments, each training full body pose435in the training pose dataset may be “real” poses in the sense that they were not generated, either in whole or in part, by the model400. In particular embodiments, each training full body pose435may be automatically obtained by retrieving different poses of a person from an pose database, wherein each of the different poses comprise a full body pose435of the person. These “real” training full body poses435serve as ground truths to train the Discriminator410network to identify which full body poses are “real” and which are “fake”. The training full body pose435may also depict a person, for example and not by way of limitation, sitting, standing, or kneeling, or in another posture that is the same or differs from the generated upper body pose205or the generated lower body pose215outputted by the Generator405. The randomness in the poses helps the trained machine-learning model400to be more robust, since in operation it may be unknown what kind of body poses the machine-learning model400would be asked to process.

In particular embodiments, training of the machine-learning model200may be performed simultaneously, or in stages. For example, a first stage may be for training the Generator405, a second stage may be for training the Discriminator410based on the outputs of the Generator405, and a third stage may be for retraining/refining the Generator405to better “fool” the trained Discriminator410. As another example, the training of Generator405and Discriminator410may occur simultaneously.

In particular embodiments, the Generator405may be configured to receive an upper body pose205and generate a temporal sequence of lower body poses215, for example, a sequence of lower body poses of a user standing, sitting, or walking. Rather than generate a single lower body pose215, the Generator405may generate a temporal sequence of lower body poses that may be combined with a corresponding upper body pose to generate a temporal sequence of full body poses. In particular embodiments the Discriminator410may be configured and trained to determine whether an inputted sequence of poses is temporally consistent, and thus discriminate between a “fake” temporal sequence of full body poses (those comprising a temporal sequence of lower body poses outputted by the Generator405) and a “real” temporal sequence of training lower body poses from a training pose database440that are not generated by Generator405. In particular embodiments, the temporal sequence of training full body poses may comprise full body poses from one or more images. In this temporal GAN the Generator405and the Discriminator410may be considered as adversaries, because the objective of the Generator405is to generate temporal sequences of fake poses that would fool the Discriminator410(in other words, to increase the Discriminator's410error rate), and the objective of the Discriminator410is to correctly distinguish “fake” temporal sequences of full body poses from the Generator405and “real” temporal sequences of full body poses. Once trained, the temporal Generator can be trained to output a realistic temporal sequence of lower body poses given an inputted upper body pose205. While this disclosure, for readability and clarity, primarily describes training and utilizing only one pose at a time using the methods described herein, the GAN may instead be trained and utilized to output a temporal sequence of lower body poses using the same methods described herein.

FIG.5illustrates an example method500for training a Generator based on a loss, in accordance with particular embodiments. At a high level, during the training the Generator405, the parameters of the Generator405may be iteratively updated based on a comparison between the generated “fake” full body pose425(which comprises lower body pose215generated by the Generator) and the corresponding training full body pose435. An objective of training is to maximize the loss460of prediction for “fake” poses. In doing so, the goal is to have the Generator405learn how to generate, based on an upper body pose205, upper body poses215that can be used to generate a “fake” full body pose425that looks sufficiently “real.”

The method may begin at step510, where a computing system may receive an upper body pose205that corresponds to a first portion of a body of user102, the first portion of the body comprising a head and arm of user102. The upper body pose205inputted into the Generator405may be an upper body pose generated based on sensor data and/or image data using one or more techniques described herein, or the upper body pose205inputted into the Generator405may be an upper body pose extracted from one or more real images of a person.

At step520, the machine-learning model may generate, by processing the upper body pose205, a lower body pose215that corresponds to a second portion of the body of user102, the second portion of the body comprising a leg of user102. In particular embodiments the Generator405may generate the lower body pose215based on the received upper body pose205. In particular embodiments the generated lower body pose215corresponds to a second portion of the body comprising a leg of the user. The generated lower body pose215may be used to generate a “fake” full body pose425that comprises the generated lower body pose215.

In particular embodiments the computing system may determine contextual information associated with the artificial reality system100. In particular embodiments, the Generator may receive the contextual information. The Generator405may utilize this contextual information for generating lower body poses. This contextual information may be associated with a particular time at which the sensor data and/or image data was captured. For example and not by way of limitation, the contextual information may comprise information about whether user102is sitting or standing. In another example, the contextual information may comprise information about an application the user is interacting with when the upper body pose is determined, for example and not by way of limitation, an artificial reality application (e.g., whether the user is interacting with application associated with a business meeting, or if they are interacting with an application associated with a game, such as a dance contest). In particular embodiments, the Generator405may receive one or more contextual information associated with the artificial reality system100. In particular embodiments, the lower body pose215is generated by further processing the contextual information using the machine-learning model.

In particular embodiments, the computing system may determine one or more physical constraints associated with an upper body pose or lower body pose, using for example, a physics aware data augmentation method. The Generator405may receive and utilize one or more of these physical constraints to generate more realistic lower body poses, or a more realistic temporal sequence of lower body poses. These physical constraints may be associated with one or more joint poses of the user, for example and not by way of limitation, a physical limitation on the possible range of motion for a knee joint pose that simulates one or more physical limitations of the human body. In particular embodiments, the computing system may determine one or more physical constraints when generating a temporal sequence of lower body poses. For example and not by way of limitation, the computing system may determine temporal limitations on the rate of acceleration or jitter of one or more joints between one or more sequential poses in a temporal sequence of poses. In particular embodiments, the Generator405may use this constraint to determine a more realistic temporal sequence of lower body poses.

At step530, the system may generate, based on the upper body pose205and the lower body pose215, a full body pose425.FIG.6illustrates generating a full body pose from a generated upper body pose and a generated lower body pose. In particular embodiments, the upper body pose215is generated using the methods described herein and corresponds to a first portion of a body of a user, the first portion of the body comprising a head and arm of a user. In particular embodiments, the lower body pose215is generated using the methods described herein and corresponds to a second portion of the body of a user, the second portion of a body comprising a leg of the user. In particular embodiments, full body pose425may generated by the machine-learning model400. In particular embodiments, full body pose425may be generated by the computing system in a post-processing step.

At step540, the system may determine, using the Discriminator410, whether the full body pose425is likely generated using the lower body pose215generated by the Generator405(i.e., whether it is “fake” or “real”). In particular embodiments, the goal of training the Discriminator is to have it learn how to correctly discriminate between “real” and “fake” full body poses. For example, for each full body pose, the Discriminator410may output a value between 0 and 1 that represents a confidence or probability/likelihood of the pose being generated by the Generator (i.e., “fake”). For instance, a value closer to 1 may indicate a higher probability/confidence that the pose is “real” (i.e., not generated by the Generator) and a value closer to 0 may indicate a lower probability/confidence that the pose is “real” (which implicitly means a higher probability/confidence that the pose is “fake”). These predictions may be compared to known labels of the poses that indicate which are “fake” and which are “real.”

In particular embodiments, the system may compute a loss based on D's determination of whether the full body pose is likely generated using the lower body pose215generated by the Generator405. For instance, the loss may be computed based on a comparison of the prediction (e.g., confidence/probability score) made by the Discriminator and the known label (which may be an implicit label) of the full body pose being “fake.” In particular embodiments, for a given pose, the prediction of the Discriminator410(represented as an output a value between 0 and 1 that represents a confidence or probability/likelihood of the pose being generated by the Generator405) may be compared to known labels of the poses that indicate which are “fake” and which are “real.” For example, if a full body pose is known to be “fake,” a prediction that is closer to 1 may result in a higher loss and a prediction that is closer to 0 may result in a lower loss. The loss, in other words, may be a measure of the correctness of the Discriminator's410predictions.

At step550, the system may then update the Generator based on the loss. The parameters of the Generator may be updated with the goal of maximizing the loss outputted by Discriminator410. Since the Generator's objective is to “fool” the Discriminator in thinking that the full body poses generated using the lower body pose215generated by the Generator are “real,” the training algorithm may be configured to optimize the loss. In other words, the Generator's parameters would be updated with the goal optimizing the Generator to generate lower body poses (that are then utilized to generate a full body pose) that would cause the loss of the Discriminator's prediction to increase (i.e., to increase its incorrect predictions that a given full body pose, generated using the lower body pose215generated by the Generator405, is unlikely generated using the Generator405).

Then at step560, the system may determine whether the training of Generator405is complete. In particular embodiments the system may determine whether the training is complete based on one or more termination criteria. For example, if the loss is below a predetermined threshold and/or if its changes over the last several iterations have stabilized (e.g., fluctuated within a predetermined range), then the system may determine that training is complete. Alternatively or additionally, training may be deemed complete if a predetermined number of training iterations have been completed or if a predetermined number of training samples have been used. In the event training is not yet complete, the system may, in particular embodiments, repeat the process starting from step510to continue training the Generator. If it is instead determined that training is complete, then the Trained Generator may be used in operation.

The training objective is for the Generator405to learn to generate poses215that would fool the Discriminator410in thinking the poses are “real.” In particular embodiments, the Generator405may generate a lower body pose215and the Discriminator410may predict a likelihood of the resulting full body pose425being “real” or “fake”450. If a high predicted value represents the Discriminator410thinking that the pose is more likely to be “real,” then the objective of the training may be expressed in terms of maximizing the Discriminator's410predicted value for the “fake” full body pose215. If the Discriminator's410prediction correctness for the “fake” full body pose215is represented as a loss460, then the objective of the training may be expressed in terms of maximizing that loss460. Based on the loss function and training objective, the parameters of the Generator405may be iteratively updated after each training so that the Generator405becomes better at generating lower body poses215that could “fool” the Discriminator410. Thus, once trained, the Generator405may be used to process a given upper body pose and automatically generate a realistic lower body pose.

Returning toFIG.4, while the Generator405is being trained, it may be used to simultaneously train the Discriminator410in accordance with particular embodiments. At a high level, the Generator405may process a given upper body pose205and generate a lower body pose215. The computing system may utilize the upper body pose205and the lower body pose215to generate a full body pose425. The generated full body pose425, along with one or more training full body poses435from a training pose database440may be provided as input to the Discriminator410.

FIG.7illustrates an example method700for training a Discriminator410. In particular embodiments, the Generator405may be used to train the Discriminator410. In particular embodiments, the Generator405and the Discriminator410may be trained simultaneously. The Discriminator410may be tasked with processing the input poses (e.g., the “fake” full body pose425, and the “real” training full body pose435) and predicting which are “real” and which are “fake”450. Conceptually, the goal of training the Discriminator410may be to maximize the Discriminator's410prediction values for the generated full body pose425and the training full body pose435, and minimize the Discriminator's410prediction value for the generated full body pose425. In other words, if the Discriminator's410prediction correctness for the “fake” pose (e.g., the generated full body pose425) and the “real” poses (i.e., the training full body pose435) are represented as losses460and465, respectively, the goal of training the Discriminator410would be to minimize the losses (i.e., minimize the incorrect predictions). Based on the loss function and training objective, the parameters of the Discriminator410may be iteratively updated after each prediction so that the Discriminator410becomes better at discriminating between “real” and “fake” full body poses.

The method may begin at step710, where a computing system may receive an upper body pose205that corresponds to a first portion of a body of user102, the first portion of the body comprising a head and arm of user102. The upper body pose205inputted into the Generator405may be an upper body pose generated based on sensor data and/or image data using one or more techniques described herein, or the upper body pose205inputted into the Generator405may be an upper body pose extracted from one or more real images of a person. In particular embodiments step710may proceed similarly to step510described herein.

At step720, the machine-learning model may generate, by processing the upper body pose205, a lower body pose215that corresponds to a second portion of the body of user102, the second portion of the body comprising a leg of user102. In particular embodiments the Generator405may generate the lower body pose215based on the received upper body pose205. In particular embodiments the generated lower body pose215corresponds to a second portion of the body comprising a leg of the user. In particular embodiments, the Generator405may generate the lower body pose215based on its current parameters, which may be iteratively updated during training so that the Generator gets better at generating realistic lower body poses. The generated lower body pose215may be used to generate a “fake” full body pose425that comprises the generated lower body pose215. In particular embodiments step720may proceed similarly to step520described herein.

In particular embodiments the computing system may determine contextual information associated with the artificial reality system100. In particular embodiments, the Generator may receive the contextual information. The Generator405may utilize this contextual information for generating lower body poses. This contextual information may be associated with a particular time at which the sensor data and/or image data was captured. For example and not by way of limitation, the contextual information may comprise information about whether user102is sitting or standing. In another example, the contextual information may comprise information about an application the user is interacting with when the upper body pose is determined, for example and not by way of limitation, an artificial reality application (e.g., whether the user is interacting with application associated with a business meeting, or if they are interacting with an application associated with a game, such as a dance contest). In particular embodiments, the Generator405may receive one or more contextual information associated with the artificial reality system100. In particular embodiments, the lower body pose215is generated by further processing the contextual information using the machine-learning model.

At step730, the system may generate, based on the upper body pose205and the lower body pose215, a full body pose425. In particular embodiments, full body pose425may generated by the machine-learning model400. In particular embodiments, full body pose425may be generated by the computing system in a post-processing step. In particular embodiments step730may proceed similarly to step530described herein.

At step740, the system may determine, using the Discriminator410, whether the full body pose425is likely generated using the lower body pose215generated by the generated using the Generator405(i.e., whether it is “fake” or “real”). In particular embodiments, the goal of training the Discriminator is to have it learn how to correctly discriminate between “real” and “fake” full body poses. For example, for each full body pose, the Discriminator410may output a value between 0 and 1 that represents a confidence or probability/likelihood of the pose being generated by the Generator (i.e., “fake”). For instance, a value closer to 1 may indicate a higher probability/confidence that the pose is “real” (i.e., not generated by the Generator) and a value closer to 0 may indicate a lower probability/confidence that the pose is “real” (which implicitly means a higher probability/confidence that the pose is “fake”). These predictions may be compared to known labels of the poses that indicate which are “fake” and which are “real.” In particular embodiments step740may proceed similarly to step540described herein.

In particular embodiments, the system may compute a loss based on the Discriminator's determination of whether the full body pose is likely generated using the lower body pose215generated by the generated by the Generator405. For instance, the loss may be computed based on a comparison of the prediction (e.g., confidence/probability score) made by the Discriminator and the known label (which may be an implicit label) of the full body pose being “fake.” In particular embodiments, for a given pose, the prediction of the Discriminator410(represented as an output a value between 0 and 1 that represents a confidence or probability/likelihood of the pose being generated by the Generator405) may be compared to known labels of the poses that indicate which are “fake” and which are “real.” For example, if a full body pose is known to be “fake,” a prediction that is closer to 1 may result in a higher loss and a prediction that is closer to 0 may result in a lower loss. The loss, in other words, may be a measure of the correctness of the Discriminator's410predictions.

At step750, the system may then update the Discriminator based on the loss. The parameters of the Discriminator410may be updated with the goal of minimizing the loss outputted by the Discriminator. In particular embodiments, these losses may be back-propagated and used by the training algorithm to update the parameters of the Discriminator410so that, over the course of the training, the Discriminator would progressively become better at discriminating between “fake” versus “real” poses. The goal of the training algorithm may be to minimize the losses (i.e., minimize the incorrect predictions).

Then at step760, the system may determine whether the training of Discriminator405is complete. In particular embodiments the system may determine whether the training is complete based on one or more termination criteria. For example, if the loss is below a predetermined threshold and/or if its changes over the last several iterations have stabilized (e.g., fluctuated within a predetermined range), then the system may determine that training is complete. Alternatively or additionally, training may be deemed complete if a predetermined number of training iterations have been completed or if a predetermined number of training samples have been used. In the event training is not yet complete, the system may, in particular embodiments, repeat the process starting from step710to continue training the Discriminator.

Although this disclosure describes and illustrates particular steps of the method ofFIG.7as occurring in a particular order, this disclosure contemplates any suitable steps of the method ofFIG.7occurring in any suitable order. Moreover, although this disclosure describes and illustrates an example method for training a Discriminator, including the particular steps of the method ofFIG.7, this disclosure contemplates any suitable method for training a Discriminator, including any suitable steps, which may include all, some, or none of the steps of the method ofFIG.7, where appropriate. Furthermore, although this disclosure describes and illustrates particular components, devices, or systems carrying out particular steps of the method ofFIG.7, this disclosure contemplates any suitable combination of any suitable components, devices, or systems carrying out any suitable steps of the method ofFIG.7.

Returning toFIG.5, once the Generator405has been trained, it can be configured to receive an input upper body pose205and generate a lower body pose215. After training, the pose would be realistic, such that when combined with the upper body pose205the computing system would generate a realistic full body pose425. Further, once the Generator is trained, it may be used to generate lower body poses based on any inputted upper body pose (in other words, it is not limited to generating lower body poses based on poses that appeared in the training pose dataset). The trained Generator may also be distributed to different platforms different from the training system, including, for example, users' mobile devices or other personal computing devices.

At step570the Trained Generator may access an upper body pose205. The upper body pose205may correspond to a first portion of a body of user102, the first portion of the body comprising a head and arm of user102. The upper body pose205inputted into the Trained Generator may be an upper body pose generated based on sensor data using one or more techniques described herein, or the upper body pose205inputted into the Trained Generator405may be an upper body pose from one or more real images of a person.

Then at step580, the Trainer Generator may generate a lower body pose that corresponds to a second portion of the body of the user. The lower body pose may correspond to a second portion of the body of user102, the second portion of the body comprising a leg of user102. In particular embodiments the Trained Generator may generate the lower body pose215based on the received upper body pose205. In particular embodiments the generated lower body pose215corresponds to a second portion of the body comprising a leg of the user.

In particular embodiments, the trained Generator may also receive and process contextual information specifying further information for generating an accurate lower body pose215. In particular embodiments the contextual information may be associated with a time at which the sensor data is captured that was used to generate upper body pose205. For example, the Generator may receive contextual information comprising an application the user102was interacting with when sensor data was captured, such as a virtual reality application used for virtual business meetings. Based on this information, the trained Generator405may be more likely to generate a lower body pose215that is typical of a user sitting or standing in a workplace setting.

Returning toFIG.6, In particular embodiments the computing system may generate based on the upper body pose205and the lower body pose215, a full body pose425. In particular embodiments, full body pose425may be generated by the machine-learning model400. In particular embodiments, full body pose425may be generated by the computing system in a post-processing step. In particular embodiments, this full body pose425may be utilized for a variety of applications. For example, computing system108may utilize fully body pose425to generate an avatar of user102in a virtual reality or artificial reality space. In particular embodiments the computing system108may only utilize a portion of the full body pose425for various applications, for example and not by way of limitation, an upper body pose (e.g., from the user's head to the user's hip) or the inferred pose of only one or more joints (e.g. an elbow joint230, or a knee joint290).

Although this disclosure describes and illustrates particular steps of the method of FIG. as occurring in a particular order, this disclosure contemplates any suitable steps of the method ofFIG.5occurring in any suitable order. Moreover, although this disclosure describes and illustrates an example method for training a generator based on encoded representations of features, including the particular steps of the method ofFIG.5, this disclosure contemplates any suitable method for training a generator, including any suitable steps, which may include all, some, or none of the steps of the method ofFIG.5, where appropriate. Furthermore, although this disclosure describes and illustrates particular components, devices, or systems carrying out particular steps of the method ofFIG.5, this disclosure contemplates any suitable combination of any suitable components, devices, or systems carrying out any suitable steps of the method of FIG.

Although this disclosure describes and illustrates particular steps of the method ofFIG.4as occurring in a particular order, this disclosure contemplates any suitable steps of the method ofFIG.4occurring in any suitable order. Moreover, although this disclosure describes and illustrates an example method for training a generator based on recurrence loss, including the particular steps of the method ofFIG.4, this disclosure contemplates any suitable method for for training a generator based on recurrence loss, including any suitable steps, which may include all, some, or none of the steps of the method ofFIG.4, where appropriate. Furthermore, although this disclosure describes and illustrates particular components, devices, or systems carrying out particular steps of the method ofFIG.4, this disclosure contemplates any suitable combination of any suitable components, devices, or systems carrying out any suitable steps of the method ofFIG.4.

FIG.8illustrates an example method800for generating a full body pose of the user based on an upper body pose and a lower body pose. The method may begin at step810, where a computing system may receive sensor data captured by one more sensors coupled to a user. In particular embodiments the one or more sensors may be coupled to user102. In particular embodiments, the one or more sensors may be associated with the headset104worn by the user. For example and not by way of limitation, the headset104may include a gyroscope or inertial measurement unit that tracks the user's real-time movements and output sensor data to represent or describe the movement. As another example and not by way of limitation, the one or more controllers106may include inertial measurement units (IMUs) and infrared (IR) light emitting diodes (LEDs) configured to collect and send IMU sensor data to the computing system108. As another example, the sensor data may comprise one or more image data from one or more components of artificial reality system100.

At step820, the computing system may generate an upper body pose that corresponds to a first portion of a body of the user, the first portion of the body comprising a head and arm of the user. In particular embodiments the computing system may utilize sensor data to generate the upper body pose. In particular embodiments, computing system108may determine a plurality of poses corresponding to a plurality of predetermined body parts of user102, for example and not by way of limitation, a joint pose (e.g., a head pose210or a wrist pose220). In particular embodiments one or more joint poses comprising the upper body pose200may be determined or inferred using a combination of one or more techniques, for example and not by way of limitation localization techniques (e.g., SLAM), machine-learning techniques (e.g., a neural network), known spatial relationships with one or more artificial reality system components (e.g., a known spatial relationship between headset104and head pose210), visualization techniques (e.g., image segmentation), or optimization techniques (e.g., a non-linear solver).

At step830, the computing system may generate a lower body pose that corresponds to a second portion of the body of the user, the second portion of the body comprising a leg of the user. In particular embodiments, the computing system may utilize a machine-learning model to generate the lower body pose, for example and not by way of limitation, a Generative Adversarial Network (GAN) comprising a Generator network and a Discriminator Network. In particular embodiments, the Generator may receive contextual information associated with the upper body pose. The Generator may utilize this contextual information for generating lower body poses. This contextual information may be associated with a particular time at which the sensor data and/or image data was captured.

At step840, the computing system may generate a full body pose of the user based on the upper body pose and the lower body pose. In particular embodiments, full body pose may be generated by the machine-learning model. In particular embodiments, full body pose may be generated by the computing system in a post-processing step. In particular embodiments, this full body pose may be utilized for a variety of applications. For example, computing system may utilize fully body pose to generate an avatar of user in a virtual reality or artificial reality space.

Particular embodiments may repeat one or more steps of the method ofFIG.8, where appropriate. Although this disclosure describes and illustrates particular steps of the method ofFIG.8as occurring in a particular order, this disclosure contemplates any suitable steps of the method ofFIG.8occurring in any suitable order. Moreover, although this disclosure describes and illustrates an example method for generating a full body pose of the user based on an upper body pose and a lower body pose, including the particular steps of the method ofFIG.8, this disclosure contemplates any suitable method for generating a full body pose of the user based on an upper body pose and a lower body pose, including any suitable steps, which may include all, some, or none of the steps of the method ofFIG.8, where appropriate. Furthermore, although this disclosure describes and illustrates particular components, devices, or systems carrying out particular steps of the method ofFIG.8, this disclosure contemplates any suitable combination of any suitable components, devices, or systems carrying out any suitable steps of the method ofFIG.8.

FIG.9illustrates an example network environment900associated with a social-networking system. Network environment900includes a client system930, a social-networking system960, and a third-party system970connected to each other by a network910. AlthoughFIG.9illustrates a particular arrangement of client system930, social-networking system960, third-party system970, and network910, this disclosure contemplates any suitable arrangement of client system930, social-networking system960, third-party system970, and network910. As an example and not by way of limitation, two or more of client system930, social-networking system960, and third-party system970may be connected to each other directly, bypassing network910. As another example, two or more of client system930, social-networking system960, and third-party system970may be physically or logically co-located with each other in whole or in part. Moreover, althoughFIG.9illustrates a particular number of client systems930, social-networking systems960, third-party systems970, and networks910, this disclosure contemplates any suitable number of client systems930, social-networking systems960, third-party systems970, and networks910. As an example and not by way of limitation, network environment900may include multiple client system930, social-networking systems960, third-party systems970, and networks910.

This disclosure contemplates any suitable network910. As an example and not by way of limitation, one or more portions of network910may include an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), a portion of the Internet, a portion of the Public Switched Telephone Network (PSTN), a cellular telephone network, or a combination of two or more of these. Network910may include one or more networks910.

Links950may connect client system930, social-networking system960, and third-party system970to communication network910or to each other. This disclosure contemplates any suitable links950. In particular embodiments, one or more links950include one or more wireline (such as for example Digital Subscriber Line (DSL) or Data Over Cable Service Interface Specification (DOCSIS)), wireless (such as for example Wi-Fi or Worldwide Interoperability for Microwave Access (WiMAX)), or optical (such as for example Synchronous Optical Network (SONET) or Synchronous Digital Hierarchy (SDH)) links. In particular embodiments, one or more links950each include an ad hoc network, an intranet, an extranet, a VPN, a LAN, a WLAN, a WAN, a WWAN, a MAN, a portion of the Internet, a portion of the PSTN, a cellular technology-based network, a satellite communications technology-based network, another link950, or a combination of two or more such links950. Links950need not necessarily be the same throughout network environment900. One or more first links950may differ in one or more respects from one or more second links950.

In particular embodiments, client system930may be an electronic device including hardware, software, or embedded logic components or a combination of two or more such components and capable of carrying out the appropriate functionalities implemented or supported by client system930. As an example and not by way of limitation, a client system930may include a computer system such as a desktop computer, notebook or laptop computer, netbook, a tablet computer, e-book reader, GPS device, camera, personal digital assistant (PDA), handheld electronic device, cellular telephone, smartphone, augmented/virtual reality device, other suitable electronic device, or any suitable combination thereof. This disclosure contemplates any suitable client systems930. A client system930may enable a network user at client system930to access network910. A client system930may enable its user to communicate with other users at other client systems930.

In particular embodiments, client system930may include a web browser932, and may have one or more add-ons, plug-ins, or other extensions. A user at client system930may enter a Uniform Resource Locator (URL) or other address directing the web browser932to a particular server (such as server962, or a server associated with a third-party system970), and the web browser932may generate a Hyper Text Transfer Protocol (HTTP) request and communicate the HTTP request to server. The server may accept the HTTP request and communicate to client system930one or more Hyper Text Markup Language (HTML) files responsive to the HTTP request. Client system930may render a webpage based on the HTML files from the server for presentation to the user. This disclosure contemplates any suitable webpage files. As an example and not by way of limitation, webpages may render from HTML files, Extensible Hyper Text Markup Language (XHTML) files, or Extensible Markup Language (XML) files, according to particular needs. Such pages may also execute scripts, combinations of markup language and scripts, and the like. Herein, reference to a webpage encompasses one or more corresponding webpage files (which a browser may use to render the webpage) and vice versa, where appropriate.

In particular embodiments, social-networking system960may be a network-addressable computing system that can host an online social network. Social-networking system960may generate, store, receive, and send social-networking data, such as, for example, user-profile data, concept-profile data, social-graph information, or other suitable data related to the online social network. Social-networking system960may be accessed by the other components of network environment900either directly or via network910. As an example and not by way of limitation, client system930may access social-networking system960using a web browser932, or a native application associated with social-networking system960(e.g., a mobile social-networking application, a messaging application, another suitable application, or any combination thereof) either directly or via network910. In particular embodiments, social-networking system960may include one or more servers962. Each server962may be a unitary server or a distributed server spanning multiple computers or multiple datacenters. Servers962may be of various types, such as, for example and without limitation, web server, news server, mail server, message server, advertising server, file server, application server, exchange server, database server, proxy server, another server suitable for performing functions or processes described herein, or any combination thereof. In particular embodiments, each server962may include hardware, software, or embedded logic components or a combination of two or more such components for carrying out the appropriate functionalities implemented or supported by server962. In particular embodiments, social-networking system960may include one or more data stores964. Data stores964may be used to store various types of information. In particular embodiments, the information stored in data stores964may be organized according to specific data structures. In particular embodiments, each data store964may be a relational, columnar, correlation, or other suitable database. Although this disclosure describes or illustrates particular types of databases, this disclosure contemplates any suitable types of databases. Particular embodiments may provide interfaces that enable a client system930, a social-networking system960, or a third-party system970to manage, retrieve, modify, add, or delete, the information stored in data store964.

In particular embodiments, social-networking system960may store one or more social graphs in one or more data stores964. In particular embodiments, a social graph may include multiple nodes—which may include multiple user nodes (each corresponding to a particular user) or multiple concept nodes (each corresponding to a particular concept)—and multiple edges connecting the nodes. Social-networking system960may provide users of the online social network the ability to communicate and interact with other users. In particular embodiments, users may join the online social network via social-networking system960and then add connections (e.g., relationships) to a number of other users of social-networking system960to whom they want to be connected. Herein, the term “friend” may refer to any other user of social-networking system960with whom a user has formed a connection, association, or relationship via social-networking system960.

In particular embodiments, social-networking system960may provide users with the ability to take actions on various types of items or objects, supported by social-networking system960. As an example and not by way of limitation, the items and objects may include groups or social networks to which users of social-networking system960may belong, events or calendar entries in which a user might be interested, computer-based applications that a user may use, transactions that allow users to buy or sell items via the service, interactions with advertisements that a user may perform, or other suitable items or objects. A user may interact with anything that is capable of being represented in social-networking system960or by an external system of third-party system970, which is separate from social-networking system960and coupled to social-networking system960via a network910.

In particular embodiments, social-networking system960may be capable of linking a variety of entities. As an example and not by way of limitation, social-networking system960may enable users to interact with each other as well as receive content from third-party systems970or other entities, or to allow users to interact with these entities through an application programming interfaces (API) or other communication channels.

In particular embodiments, a third-party system970may include one or more types of servers, one or more data stores, one or more interfaces, including but not limited to APIs, one or more web services, one or more content sources, one or more networks, or any other suitable components, e.g., that servers may communicate with. A third-party system970may be operated by a different entity from an entity operating social-networking system960. In particular embodiments, however, social-networking system960and third-party systems970may operate in conjunction with each other to provide social-networking services to users of social-networking system960or third-party systems970. In this sense, social-networking system960may provide a platform, or backbone, which other systems, such as third-party systems970, may use to provide social-networking services and functionality to users across the Internet.

In particular embodiments, a third-party system970may include a third-party content object provider. A third-party content object provider may include one or more sources of content objects, which may be communicated to a client system930. As an example and not by way of limitation, content objects may include information regarding things or activities of interest to the user, such as, for example, movie show times, movie reviews, restaurant reviews, restaurant menus, product information and reviews, or other suitable information. As another example and not by way of limitation, content objects may include incentive content objects, such as coupons, discount tickets, gift certificates, or other suitable incentive objects.

In particular embodiments, social-networking system960also includes user-generated content objects, which may enhance a user's interactions with social-networking system960. User-generated content may include anything a user can add, upload, send, or “post” to social-networking system960. As an example and not by way of limitation, a user communicates posts to social-networking system960from a client system930. Posts may include data such as status updates or other textual data, location information, photos, videos, links, music or other similar data or media. Content may also be added to social-networking system960by a third-party through a “communication channel,” such as a newsfeed or stream.

In particular embodiments, social-networking system960may include a variety of servers, sub-systems, programs, modules, logs, and data stores. In particular embodiments, social-networking system960may include one or more of the following: a web server, action logger, API-request server, relevance-and-ranking engine, content-object classifier, notification controller, action log, third-party-content-object-exposure log, inference module, authorization/privacy server, search module, advertisement-targeting module, user-interface module, user-profile store, connection store, third-party content store, or location store. Social-networking system960may also include suitable components such as network interfaces, security mechanisms, load balancers, failover servers, management-and-network-operations consoles, other suitable components, or any suitable combination thereof. In particular embodiments, social-networking system960may include one or more user-profile stores for storing user profiles. A user profile may include, for example, biographic information, demographic information, behavioral information, social information, or other types of descriptive information, such as work experience, educational history, hobbies or preferences, interests, affinities, or location. Interest information may include interests related to one or more categories. Categories may be general or specific. As an example and not by way of limitation, if a user “likes” an article about a brand of shoes the category may be the brand, or the general category of “shoes” or “clothing.” A connection store may be used for storing connection information about users. The connection information may indicate users who have similar or common work experience, group memberships, hobbies, educational history, or are in any way related or share common attributes. The connection information may also include user-defined connections between different users and content (both internal and external). A web server may be used for linking social-networking system960to one or more client systems930or one or more third-party system970via network910. The web server may include a mail server or other messaging functionality for receiving and routing messages between social-networking system960and one or more client systems930. An API-request server may allow a third-party system970to access information from social-networking system960by calling one or more APIs. An action logger may be used to receive communications from a web server about a user's actions on or off social-networking system960. In conjunction with the action log, a third-party-content-object log may be maintained of user exposures to third-party-content objects. A notification controller may provide information regarding content objects to a client system930. Information may be pushed to a client system930as notifications, or information may be pulled from client system930responsive to a request received from client system930. Authorization servers may be used to enforce one or more privacy settings of the users of social-networking system960. A privacy setting of a user determines how particular information associated with a user can be shared. The authorization server may allow users to opt in to or opt out of having their actions logged by social-networking system960or shared with other systems (e.g., third-party system970), such as, for example, by setting appropriate privacy settings. Third-party-content-object stores may be used to store content objects received from third parties, such as a third-party system970. Location stores may be used for storing location information received from client systems930associated with users. Advertisement-pricing modules may combine social information, the current time, location information, or other suitable information to provide relevant advertisements, in the form of notifications, to a user.

FIG.10illustrates an example computer system1000. In particular embodiments, one or more computer systems1000perform one or more steps of one or more methods described or illustrated herein. In particular embodiments, one or more computer systems1000provide functionality described or illustrated herein. In particular embodiments, software running on one or more computer systems1000performs one or more steps of one or more methods described or illustrated herein or provides functionality described or illustrated herein. Particular embodiments include one or more portions of one or more computer systems1000. Herein, reference to a computer system may encompass a computing device, and vice versa, where appropriate. Moreover, reference to a computer system may encompass one or more computer systems, where appropriate.

This disclosure contemplates any suitable number of computer systems1000. This disclosure contemplates computer system1000taking any suitable physical form. As example and not by way of limitation, computer system1000may be an embedded computer system, a system-on-chip (SOC), a single-board computer system (SBC) (such as, for example, a computer-on-module (COM) or system-on-module (SOM)), a desktop computer system, a laptop or notebook computer system, an interactive kiosk, a mainframe, a mesh of computer systems, a mobile telephone, a personal digital assistant (PDA), a server, a tablet computer system, an augmented/virtual reality device, or a combination of two or more of these. Where appropriate, computer system1000may include one or more computer systems1000; be unitary or distributed; span multiple locations; span multiple machines; span multiple data centers; or reside in a cloud, which may include one or more cloud components in one or more networks. Where appropriate, one or more computer systems1000may perform without substantial spatial or temporal limitation one or more steps of one or more methods described or illustrated herein. As an example and not by way of limitation, one or more computer systems1000may perform in real time or in batch mode one or more steps of one or more methods described or illustrated herein. One or more computer systems1000may perform at different times or at different locations one or more steps of one or more methods described or illustrated herein, where appropriate.

In particular embodiments, computer system1000includes a processor1002, memory1004, storage1006, an input/output (I/O) interface1008, a communication interface1010, and a bus1012. Although this disclosure describes and illustrates a particular computer system having a particular number of particular components in a particular arrangement, this disclosure contemplates any suitable computer system having any suitable number of any suitable components in any suitable arrangement.

In particular embodiments, processor1002includes hardware for executing instructions, such as those making up a computer program. As an example and not by way of limitation, to execute instructions, processor1002may retrieve (or fetch) the instructions from an internal register, an internal cache, memory1004, or storage1006; decode and execute them; and then write one or more results to an internal register, an internal cache, memory1004, or storage1006. In particular embodiments, processor1002may include one or more internal caches for data, instructions, or addresses. This disclosure contemplates processor1002including any suitable number of any suitable internal caches, where appropriate. As an example and not by way of limitation, processor1002may include one or more instruction caches, one or more data caches, and one or more translation lookaside buffers (TLBs). Instructions in the instruction caches may be copies of instructions in memory1004or storage1006, and the instruction caches may speed up retrieval of those instructions by processor1002. Data in the data caches may be copies of data in memory1004or storage1006for instructions executing at processor1002to operate on; the results of previous instructions executed at processor1002for access by subsequent instructions executing at processor1002or for writing to memory1004or storage1006; or other suitable data. The data caches may speed up read or write operations by processor1002. The TLBs may speed up virtual-address translation for processor1002. In particular embodiments, processor1002may include one or more internal registers for data, instructions, or addresses. This disclosure contemplates processor1002including any suitable number of any suitable internal registers, where appropriate. Where appropriate, processor1002may include one or more arithmetic logic units (ALUs); be a multi-core processor; or include one or more processors1002. Although this disclosure describes and illustrates a particular processor, this disclosure contemplates any suitable processor.

In particular embodiments, memory1004includes main memory for storing instructions for processor1002to execute or data for processor1002to operate on. As an example and not by way of limitation, computer system1000may load instructions from storage1006or another source (such as, for example, another computer system1000) to memory1004. Processor1002may then load the instructions from memory1004to an internal register or internal cache. To execute the instructions, processor1002may retrieve the instructions from the internal register or internal cache and decode them. During or after execution of the instructions, processor1002may write one or more results (which may be intermediate or final results) to the internal register or internal cache. Processor1002may then write one or more of those results to memory1004. In particular embodiments, processor1002executes only instructions in one or more internal registers or internal caches or in memory1004(as opposed to storage1006or elsewhere) and operates only on data in one or more internal registers or internal caches or in memory1004(as opposed to storage1006or elsewhere). One or more memory buses (which may each include an address bus and a data bus) may couple processor1002to memory1004. Bus1012may include one or more memory buses, as described below. In particular embodiments, one or more memory management units (MMUs) reside between processor1002and memory1004and facilitate accesses to memory1004requested by processor1002. In particular embodiments, memory1004includes random access memory (RAM). This RAM may be volatile memory, where appropriate. Where appropriate, this RAM may be dynamic RAM (DRAM) or static RAM (SRAM). Moreover, where appropriate, this RAM may be single-ported or multi-ported RAM. This disclosure contemplates any suitable RAM. Memory1004may include one or more memories1004, where appropriate. Although this disclosure describes and illustrates particular memory, this disclosure contemplates any suitable memory.

In particular embodiments, storage1006includes mass storage for data or instructions. As an example and not by way of limitation, storage1006may include a hard disk drive (HDD), a floppy disk drive, flash memory, an optical disc, a magneto-optical disc, magnetic tape, or a Universal Serial Bus (USB) drive or a combination of two or more of these. Storage1006may include removable or non-removable (or fixed) media, where appropriate. Storage1006may be internal or external to computer system1000, where appropriate. In particular embodiments, storage1006is non-volatile, solid-state memory. In particular embodiments, storage1006includes read-only memory (ROM). Where appropriate, this ROM may be mask-programmed ROM, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), electrically alterable ROM (EAROM), or flash memory or a combination of two or more of these. This disclosure contemplates mass storage1006taking any suitable physical form. Storage1006may include one or more storage control units facilitating communication between processor1002and storage1006, where appropriate. Where appropriate, storage1006may include one or more storages1006. Although this disclosure describes and illustrates particular storage, this disclosure contemplates any suitable storage.

In particular embodiments, I/O interface1008includes hardware, software, or both, providing one or more interfaces for communication between computer system1000and one or more I/O devices. Computer system1000may include one or more of these I/O devices, where appropriate. One or more of these I/O devices may enable communication between a person and computer system1000. As an example and not by way of limitation, an I/O device may include a keyboard, keypad, microphone, monitor, mouse, printer, scanner, speaker, still camera, stylus, tablet, touch screen, trackball, video camera, another suitable I/O device or a combination of two or more of these. An I/O device may include one or more sensors. This disclosure contemplates any suitable I/O devices and any suitable I/O interfaces1008for them. Where appropriate, I/O interface1008may include one or more device or software drivers enabling processor1002to drive one or more of these I/O devices. I/O interface1008may include one or more I/O interfaces1008, where appropriate. Although this disclosure describes and illustrates a particular I/O interface, this disclosure contemplates any suitable I/O interface.

In particular embodiments, communication interface1010includes hardware, software, or both providing one or more interfaces for communication (such as, for example, packet-based communication) between computer system1000and one or more other computer systems1000or one or more networks. As an example and not by way of limitation, communication interface1010may include a network interface controller (NIC) or network adapter for communicating with an Ethernet or other wire-based network or a wireless NIC (WNIC) or wireless adapter for communicating with a wireless network, such as a WI-FI network. This disclosure contemplates any suitable network and any suitable communication interface1010for it. As an example and not by way of limitation, computer system1000may communicate with an ad hoc network, a personal area network (PAN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), or one or more portions of the Internet or a combination of two or more of these. One or more portions of one or more of these networks may be wired or wireless. As an example, computer system1000may communicate with a wireless PAN (WPAN) (such as, for example, a BLUETOOTH WPAN), a WI-FI network, a WI-MAX network, a cellular telephone network (such as, for example, a Global System for Mobile Communications (GSM) network), or other suitable wireless network or a combination of two or more of these. Computer system1000may include any suitable communication interface1010for any of these networks, where appropriate. Communication interface1010may include one or more communication interfaces1010, where appropriate. Although this disclosure describes and illustrates a particular communication interface, this disclosure contemplates any suitable communication interface.

In particular embodiments, bus1012includes hardware, software, or both coupling components of computer system1000to each other. As an example and not by way of limitation, bus1012may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a front-side bus (FSB), a HYPERTRANSPORT (HT) interconnect, an Industry Standard Architecture (ISA) bus, an INFINIBAND interconnect, a low-pin-count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCIe) bus, a serial advanced technology attachment (SATA) bus, a Video Electronics Standards Association local (VLB) bus, or another suitable bus or a combination of two or more of these. Bus1012may include one or more buses1012, where appropriate. Although this disclosure describes and illustrates a particular bus, this disclosure contemplates any suitable bus or interconnect.

Herein, a computer-readable non-transitory storage medium or media may include one or more semiconductor-based or other integrated circuits (ICs) (such, as for example, field-programmable gate arrays (FPGAs) or application-specific ICs (ASICs)), hard disk drives (HDDs), hybrid hard drives (HHDs), optical discs, optical disc drives (ODDs), magneto-optical discs, magneto-optical drives, floppy diskettes, floppy disk drives (FDDs), magnetic tapes, solid-state drives (SSDs), RAM-drives, SECURE DIGITAL cards or drives, any other suitable computer-readable non-transitory storage media, or any suitable combination of two or more of these, where appropriate. A computer-readable non-transitory storage medium may be volatile, non-volatile, or a combination of volatile and non-volatile, where appropriate.

Herein, “or” is inclusive and not exclusive, unless expressly indicated otherwise or indicated otherwise by context. Therefore, herein, “A or B” means “A, B, or both,” unless expressly indicated otherwise or indicated otherwise by context. Moreover, “and” is both joint and several, unless expressly indicated otherwise or indicated otherwise by context. Therefore, herein, “A and B” means “A and B, jointly or severally,” unless expressly indicated otherwise or indicated otherwise by context.

The scope of this disclosure encompasses all changes, substitutions, variations, alterations, and modifications to the example embodiments described or illustrated herein that a person having ordinary skill in the art would comprehend. The scope of this disclosure is not limited to the example embodiments described or illustrated herein. Moreover, although this disclosure describes and illustrates respective embodiments herein as including particular components, elements, feature, functions, operations, or steps, any of these embodiments may include any combination or permutation of any of the components, elements, features, functions, operations, or steps described or illustrated anywhere herein that a person having ordinary skill in the art would comprehend. Furthermore, reference in the appended claims to an apparatus or system or a component of an apparatus or system being adapted to, arranged to, capable of, configured to, enabled to, operable to, or operative to perform a particular function encompasses that apparatus, system, component, whether or not it or that particular function is activated, turned on, or unlocked, as long as that apparatus, system, or component is so adapted, arranged, capable, configured, enabled, operable, or operative. Additionally, although this disclosure describes or illustrates particular embodiments as providing particular advantages, particular embodiments may provide none, some, or all of these advantages.