GENERATE A SIMPLIFIED VERSION OF A USER-GENERATED DIGITAL OBJECT

The present technology generates a simplified version of a complex avatar by capturing images and 3-D volume information of segments of the complex avatar while the complex avatar is rendered. When the complex avatar is requested in an environment in which it is not desirable to display the complex avatar, the captured images and 3-D volume information can be used to provide a simplified version of the avatar. The simplified version of the avatar can have a similar visual appearance but can be easier to render. However, the present technology permits the user with the complex avatar to continue to have approximately the same visual appearance while avoiding the degraded performance on systems not capable of rendering the complex avatar quickly enough.

BACKGROUND

Users of computing systems utilize avatars to stand in for their physical presence in a variety of applications ranging from simple chat applications to elaborate three-dimensional (3D) environments used in video game applications and virtual reality applications. A simple version of an avatar could be a shape of the shoulders and a head without any distinguishing features. Other avatars can include animated versions of a being. These avatars do not need to be in a humanoid form. Some avatars can be complex and can be associated with detailed graphics, and textures, and can be capable of various animations. As avatars become more elaborate, so too does the computing power to render the avatar.

DETAILED DESCRIPTION

Some avatars can be complex and associated with detailed graphics, textures, scripts, and they can be capable of various animations. As avatars become more elaborate, so does the computing power required to render the avatar.

Generally, more complex avatars are created with a specific environment in mind, and as such, rendering an avatar is often not so computationally intensive that a computing system that can render the environment in which the avatar will interact cannot also render the complex avatar.

However, this assumption quickly loses relevance for at least two reasons. First, rendering a 3-D environment used to be limited to specialized gaming systems or computing systems with sufficient graphics hardware such as a dedicated video card with graphics processing units and dedicated memory, graphics accelerators, etc. Now, 3-D environments are often rendered on mobile devices. This is especially true in the context of virtual reality (VR) applications that are often rendered on VR headsets. Some of these VR headsets, like the OCULUS QUEST, have only as many graphics capabilities as a mobile phone. Avatars created with the expectation that they will only be rendered on a computing system with sufficient graphics hardware can require more resources to render than might be available on other platforms.

Second, complex avatars used to be application or platform-specific. An avatar might be generated for use in a particular game or with a particular gaming system. However, as the metaverse becomes a commercial reality, the ability exists to take an avatar built for one environment into other environments, and some of these environments run on hardware that is not well suited to rendering complex avatars.

The consequences of introducing a more complex avatar than the computing system can easily render can be degraded performance in rendering the environment. This consequence can be even greater in an environment where many avatars can interact together. If several avatars require too many computing resources, the environment will have unacceptable latency.

Another consequence could be that the complex avatar might not render at all. In such an instance, the computing environment might use a generic replacement avatar rather than anything that looks like the intended avatar. This is a consequence that users would not be happy to endure. A person who has created a sophisticated avatar is likely an advanced user who would not like to be viewed as a user who is so new as not to have their own unique avatar. Further, just as a person's appearance can be used to identify the person in the real world, so too, a user's avatar can be used to identify the person in a virtual world. It is not a good user experience for a game or platform to substitute a generic avatar in place of a user's unique avatar.

One concrete example of an environment in which complex avatars result in negative consequences is a gaming platform accessible on many platforms that can range from powerful, personal computers to less capable VR headsets. The gaming platform is capable of allowing users to explore and socialize in 3-D worlds. The users are all represented by avatars. When a complex avatar is present in a world that includes a user accessing the world from one of the less capable VR headsets, the user on the VR headset might experience degraded performance or might not be able to render the complex avatar at all. Even users accessing the platform using powerful personal computers could experience degraded performance if too many complex avatars are present in the same world.

The present technology alleviates the above problems by selectively representing a simplified version of the complex avatar. The simplified version of the avatar can have a similar visual appearance but can be easier to render. While the appearance will be similar, it may lack some details, or it may lack some abilities governed by custom scripts associated with the avatar. However, the present technology permits the user with the complex avatar to continue to have approximately the same visual appearance while avoiding the degraded performance addressed above.

The present technology is not limited to avatars. Similar principles to those described herein can be adapted to generate and render a simplified version of any user-generated digital object.

The present technology is particularly well suited to environments with near real-time graphics such that dynamic objects need to be rendered and re-rendered for substantially every frame.

The present method and system generates a simplified version of a complex avatar by capturing images and 3-D volume information of segments of the complex avatar while the complex avatar is rendered. When the complex avatar is requested in an environment in which it is not desirable to display the complex avatar, the captured images and 3-D volume information can be used to provide a simplified version of the avatar.

In some embodiments, the present technology can be useful when the system that would benefit from a simplified version of a complex avatar does not have any control over the model of the complex avatar. Since the system does not have control over the model of the complex avatar, the system does not have any ability to change or improve properties associated with the complex avatar. Instead, the present technology applies to any avatar or user object that can be downloaded and rendered in a suitable computing environment.

FIG.1illustrates an example virtual world platform102for playing and hosting a multiplayer virtual reality (VR) experience that is suited to carrying out the present technology. The virtual world platform102can connect clients104through web services110and networking services112to socially interact together in a virtual world hosted by virtual world platform102.

The virtual world platform102primarily includes a client104, which is an instance of an application executed on a client device106. The client104interacts over a network connection with web services110which supports client104by providing various services through one or more application programming interfaces (APIs). A few of the main services provided by web services110are related to supporting virtual worlds through the worlds API128, user profiles through the users API132, trust and safety through the trust API144, and complex avatars through avatars API136. Web services110generally stores and provides long-term state information among other functions.

The client104also interacts with networking services112, which provides communication services between client104, networking services112, and a remote instance of client104(not shown) to share state information among respective instances of client104. In particular, state information is received from a plurality of instances of client104by networking services112as each instance of client104controls its local player116. Networking services112can transfer state information about respective players to other instances of client104when the local players116for the respective client instances are all engaged in gameplay in the same virtual world. The networking services112provide optimized packet routing through optimized packet routing service140and moderation between one or more clients through moderation service142.

The client104is the runtime environment executing on a particular client device106. While the present description sometimes refers to client104, local client, and remote clients, all are instances of the client104executing on a respective client device106. One particular user account is logged into a particular instance of client104. A local client and remote client are distinguished to illustrate how client104handles first person inputs from a user of the client device106upon which client104is executing and handles third party inputs received from another user operating their client device upon which the remote client is executing.

Client device106can be any computing device. While client104is particularly adapted to providing an immersive virtual reality experience through interactions that require a VR headset to experience, client104can also be run by computers and mobile devices. Some virtual worlds or complex avatars might not be configured to perform well on certain device types, and therefore, while client104can operate on many platforms and devices, not all virtual worlds or complex avatars will be available or have full functionality on all client devices106.

User interface service108is one service that is part of client104. User interface service108is configured to provide various user interface elements such as menus that display various user settings, available worlds, saved complex avatars, friends lists, etc. User interface service108can populate its menus through interaction with one or more APIs provided by web services110, while other portions of menus are loaded directly from user interface service108.

User interface service108can provide a menu of available worlds by calling worlds API128to retrieve a list of worlds to which the user account logged into client104is permitted to enter. Worlds API128can retrieve all public worlds from the world assets database130and send a list of those to client104. Additionally, worlds API128can request world IDs for any private worlds associated with the user account logged into client104and retrieve the private worlds from the world assets database130to send to client104. User interface service108can receive user inputs through a hardware interface to navigate through the worlds menu and to receive a selection of a world to visit.

Another user interface provided by user interface service108pertains to various user settings. Such settings can pertain to whether the human player is sitting or standing, settings to minimize motion sickness in players that are susceptible to motion sickness when playing in VR, settings to select a complex avatar, settings about how a player might be viewed and by whom a player might be viewed in a virtual world.

One notable user interface provided by the user interface service108is the trust and safety menu. User interface service108can contact users API132to retrieve current trust and safety settings from user profiles database134and display these settings in the trust and safety menu. The trust and safety menu provides the user account with the ability to determine which remote players124can see the user's avatar (local player116) or be seen by the user's avatar when they are both in the same world. For example, it may be desirable to avoid interacting with newer users of the virtual world platform102since they have not built up trust within the virtual world platform102. It may also be desirable to limit the features of a remote player's avatar that will be processed by the instance of client104to which the local user is logged in. This is because some avatars may have malicious data embedded, or the avatars may be too complex to render without degrading the performance of client device106. For example, a user account might decide to turn off lights on remote avatars to avoid shaders, disallow custom animations, etc. In some embodiments, each of these options might be set based on how trusted the remote player is. For example, a user account might allow their friend's avatars to have full features, while others only display basic avatar features.

The user interface service108can also provide options to mute or block specific remote players. Additionally, the user interface service108can provide a panic mode to audio-and-visually mute anybody who is not a friend.

After a user has selected a virtual world from the menu provided by the user interface service108, client104can download an instance of the virtual world by calling the worlds API128, which can retrieve the virtual world from worlds world assets database130and send it to client104for execution.

The world assets are large binary files built for a game engine, such as UNITY using an editor with a software development kit (SDK) provided for use with the virtual world platform102. If a user travels into a world, they need to download that world asset from world assets database130. If there are already people in that instance of the world, client104also needs a list of the avatars of those people so that the avatars can be rendered in the instance of the virtual world.

In some embodiments, a function of the worlds API128can confirm that the user account can access the requested world. While the user account should only have the ability to view public worlds in the user interface menu or should only have knowledge of links to worlds that have been shared with the user account, the worlds API128can confirm the user account is permitted to access the virtual world as a redundancy measure.

In addition to downloading the instance of the virtual world, the client104can also establish a session with networking services112for the specific instance of the world. Networking services112can provide information about the current state of the instance of the virtual world. For example, networking services112can provide a list of remote avatars126present in the virtual world instance to client104. In turn, client104can contact the avatars API136to download complex avatar assets for the list of remote complex avatars from avatar assets database138.

If the client104does not have assets for the local avatar118, client104can also contact the avatars API136to request and receive the local avatar assets. Avatar assets are a single binary file that contains all of the textures and models and animation data needed to render the avatar. In some instances, more complicated features can be included such as data about particle systems or light sources, or if the avatar should obey or defy laws of physics established in a virtual world, or if the avatar has non-standard movement dynamics.

The downloaded instance of the virtual world can be executed by client104as current world120. Current world120can include coordinates within the current world120where the local player116and each remote player124are located. The local player116and remote player124are each collision volumes of space that the respective local player116or remote player124occupy.

The local avatar118can be mapped to the local player116, and the respective remote avatar126can be mapped to their respective remote player124, thereby allowing each player to appear as their avatar in the current world120. Movements of the remote avatars126are handled by receiving state data about a respective remote avatar/player and rendering the movement or audio by client104.

The VR tracking service114pertains to clients104operating on a client device106that have access to VR tracking peripherals. For example, some VR headsets have cameras (integrated or external) to track the limbs of players. Many VR headsets can pair with controllers that can report the locations of a user's hands in space. Some client devices106include other peripherals configured to perform full skeleton tracking. VR tracking service114can fuse all VR inputs connected to the client.

The VR tracking service114can map the fused VR inputs to the local player116to allow the local player116to interact in and with the current world120. Meanwhile, the local player116can interact with the local avatar118to map the local avatar118to the local player and make the local player116appear as their avatar.

In some embodiments, there is diversity in what parts of a user's body are tracked by VR tracking service114. While some users might have full skeleton tracking, many users may only have the ability to perform hand tracking. To accommodate this disparity in hardware abilities of possible client devices106, local player116can derive portions of a skeleton that are not tracked by VR tracking service114. For example, if VR tracking service114only provides information about hand tracking for a user, the local player can still derive a full skeleton for the user and make portions of the skeleton move to accommodate the movement of the hands. In this way, an avatar's hands are not moving in a way that is disembodied from the rest of the avatar.

The local player116is the entity that moves around the environment in the current world120. It can pick things up and put them down. It does not have any animation and is a collision volume. It can do everything in the world, but it has no appearance and does not need to animate.

The local player is further connected to the networking layer, illustrated as the runtime networking service122, to broadcast state information about the local player116over the network to other users in the current world120instance.

The local player116and the remote player124are similar in that they are collision volumes that move around the environment in the current world120. The main difference is that the local player116is controlled by client104, and the user of client104is authoring the experience. In contrast, the remote player124is a playback mechanism representing actions being broadcast to the client104representing other players present in the current world120.

As addressed above, the local avatar118is overlaid with the local player116to give the user a visual appearance. Actions by the local player116are animated as the local player interacts with the current world. For example, while the local player116can interact to pick up an object in the current world120, without the local avatar118, the object would appear to float in the air. With the local avatar118overlaid the local player116, the object now appears to be held by the hand of the avatar.

The remote player124and remote avatar126work similarly to their local counterparts except for where the inputs that control the remote player124come from. The remote player124and remote avatar126are playback devices for state information received by the runtime networking service122from networking services112. WhileFIG.1only depicts one remote player124and remote avatar126, there can be many.

The current world120also has features that require networking. The current world120could have objects, like scissors or a light switch, that a user can pick up, and the object needs to broadcast its state across the network so that other users in the current world120can view the current state of the object.

Each of the local player116, current world120, and remote player124are connected to the runtime networking service122. The local player116primarily transmits updated state information for the local player116to remote instances of client104that are also executing the same virtual world. The current world120can transmit and receive state information about the instance of the virtual world. The current world executing on client104transmits state information when the state change is owned by the local player116and receives state information when the state change is owned by the remote player124.

Networking services112are the network-side part of the networking layer of the virtual world platform102. In some embodiments, portions of the networking services112are provided by a networking plug-in such as the PHOTON networking engine, which broadcasts state information to all users in an instance of a virtual world.

In addition to the general broadcasting of state information to all users interacting with an instance of a virtual world, the optimized packet routing service140provides more advanced features that provide an enhanced user experience and enforces other virtual world platform102properties, such as trust and safety configurations.

For example, to provide an enhanced user experience, the optimized packet routing service140can filter out voice packets coming from a remote player124that might be far from the local player116in the instance of the current world120. Without such optimization, remote players124that are not interacting or even visible to the local player might receive audio packets from tens or even hundreds of remote players124that would make it hard to communicate with any subsets of remote players124.

In another example, the optimized packet routing service140can enforce trust and safety configurations. As addressed above, trust and safety configurations can specify specific user accounts or groups of user accounts to be filtered so that they cannot interact with the local player116or have limited interactions with the local player116. The optimized packet routing service140can call trust API144to learn of a list of remote players124that might need to be subject to some level of filtering or blocking of network traffic going to or coming from the client104for the local player116having the trust and safety configurations.

The trust API144can determine which remote players124should be blocked for the local player116or which remote players124should have aspects of their complex avatar limited. Some of these determinations are based on logic and rules that categorize remote players124based on quantities and types of past interactions with the virtual worlds platform102. Trust API144may make these determinations by using settings stored in the user profile of the local player116and comparing these settings to data stored in user profiles of remote players124.

Another of the networking services112is a moderation service142that can provide conflict resolutions and access control. For example, before a user accesses a world, especially a private world, moderation service142can call the worlds API128to ensure the user can enter the world. In another example, there can be instances where two different users attempt to claim control of an object in a virtual world at approximately the same time. The moderation service142can handle those sorts of conflicts by selecting a particular user to control an object until they relinquish the control of the object, which allows another user to claim control of the object. A user that has control of the object can broadcast packets informing remote players124of the state of that object.

In some embodiments, client104, virtual worlds, and complex avatars can be configured to operate in a particular game engine, especially a game engine that supports three-dimensional (3D) environments. Two common game engines include UNITY and UNREAL ENGINE.

In some embodiments, to be supported by virtual world platform102, virtual worlds and complex avatars need to be developed in compliance with a software development kit (SDK). For example, complex avatars require a particular script to be usable in the virtual world platform102. In another example, there can be a number of requirements that need to be followed to get the animations of an avatar to play. In some embodiments, the SDK can define other necessary details to support particular client devices. For example, the SDK can define specific shaders to be used if the avatar is to be used on the OCULUS QUEST VR headset.

In some embodiments, the SDK requires virtual worlds to utilize a particular coding language to ensure the world has compliant behaviors. For example, the SDK can require that behaviors in worlds are defined using UDON, a programming language specific to a particular virtual world platform102, VRCHAT. In some embodiments, the programming language facilitates a world built using the programming language to comply with file access safeguards provided by the virtual world platform102. For example, a world cannot read or write anything to a hard drive, and only approved web pages can be rendered in a world on the virtual world platform102.

In some embodiments virtual world platform102can also include a simplified avatars service146. As will be described herein, simplified avatars service146can create simplified versions of complex avatars and store the avatar assets for the simplified versions of the complex avatars in avatar assets database138.

While the virtual world platform102is suited to carrying out the present technology, persons of ordinary skill in the art will appreciate that the present technology can be used in other environments.

FIG.2illustrates an example quick menu202in accordance with some aspects of the present technology. In particular, the quick menu202can be surfaced by the user interface service108on client104at any time or place in the virtual world platform102.

The quick menu202includes a quick links204section that includes many commonly used menu options such as menus to browse worlds, avatars, friends, and a safety menu208to set safety settings for the user's profile.

The trust and safety menu208provides the user account with the ability to determine which remote players124can see the user's avatar (local player116) or be seen by the user's avatar when they are both in the same world. For example, it may be desirable to avoid interacting with newer users of the virtual world platform102since they have not built up trust within the virtual world platform102. It may also be desirable to limit the features of a remote player's avatar that will be processed by the instance of client104to which the local user is logged in. This is because some avatars may have malicious data embedded, or the avatars may be too complex to render without degrading the performance of client device106. For example, a user account might decide to turn off lights on remote avatars to avoid shaders, disallow custom animations, etc. In some embodiments, each of these options might be set based on how trusted the remote player is. For example, a user account might allow their friend's avatars to have full features, while others only display basic avatar features.

The user interface service108can also provide options to mute or block specific remote players. Additionally, the user interface service108can provide a panic or safe mode210to audio-and-visually mute anybody who is not a friend.

The quick menu202can also include a quick actions206section to provide frequently used actions in a convenient location. Some example quick actions include an action to go to your homeworld, to respawn in the last world you were in, to select another user's avatar (to communicate privately, to block the user from viewing or speaking to the local player116, to copy the avatar or other function), and to select emojis.

The quick menu202also includes a dock212, which also provides access to some common features like a virtual camera, volume settings, and a settings menu, among other features.

FIG.3illustrates an example avatar302illustrated along with its skeleton304. The skeleton304is comprised of a plurality of joints. Between each joint or between a joint and the terminal portions of the avatar are segments of the avatar. For example, the upper left arm between the shoulder and elbow is a segment306, and the right hand between the wrist and the terminal portion (fingertips) is a segment308. Just as with a human skeleton, the joints are areas of articulation which are used to move the body segments.

In addition to the skeleton304, the avatar is made up of a 3-dimensional (3D) model that includes a mesh that gives the avatar a 3D shape and various artistic properties, including textures, shading, etc.

As illustrated inFIG.3, the avatar302has a humanoid form arranged in a t-shaped pose. The t-shaped pose provides a large number of unobstructed views of segments of the avatar from many angles.

As introduced above, the present technology pertains to creating and rendering a simplified version of a complex avatar. In some embodiments, the complex avatar is made up of a high number of polygons, complicated mesh patterns, custom animations, metadata, scripts, shader properties, etc. All of these attributes make complex avatars difficult and slow to render. However, regardless of the specific properties of a complex avatar, as used herein, a “complex avatar” refers to an avatar that is/has been determined to be difficult to render in a particular environment in which the complex avatar has been requested.

FIG.4Aillustrates an example method for creating a simplified version of a complex avatar. Although the example method depicts a particular sequence of operations, the sequence may be altered without departing from the scope of the present disclosure. For example, some of the operations depicted may be performed in parallel or in a different sequence that does not materially affect the function of the method. In other examples, different components of an example device or system that implements the method may perform functions at substantially the same time or in a specific sequence.

According to some embodiments, the method includes determining that an avatar is a complex avatar at block402. For example, the simplified avatars service146illustrated inFIG.1can analyze an avatar stored in avatar assets database138to determine that it is a complex avatar. A “complex avatar” refers to an avatar that may be difficult to render in a particular environment. In some embodiments, the simplified avatars service146can determine that avatar is a complex avatar when the avatar assets are uploaded for storage at the avatar assets database138. In some embodiments, the simplified avatars service146can determine that the avatar is a complex avatar after the avatar assets are downloaded by a client104and performs poorly when trying to render the complex avatar.

According to some embodiments, the method includes rendering the complex avatar at block404. For example, the simplified avatars service146can render the complex avatar. In some embodiments, the simplified avatars service146can render the complex avatar in a process in a cloud datacenter such as the datacenter used by web services110. In some embodiments, the rendered version of the complex avatar is rendered in a t-pose. In some embodiments, the complex avatar is a humanoid complex avatar defined by a humanoid tree of joints.

According to some embodiments, the method for creating a simplified version of a complex avatar includes isolating segments of a complex avatar from the plurality of segments of a rendered version of the complex avatar at block406. For example, the simplified avatars service146may isolate segments of the complex avatar from the plurality of segments of a rendered version of the complex avatar. The isolated segments of a humanoid avatar can include feet, hands, forearms, upper arms, heads, upper torso, lower torso, upper leg, lower leg, etc. Other segments are also possible, especially in avatars having additional joints or that are non-humanoid.

In order to isolate the segments of the complex avatar, the simplified avatars service146can select a segment of the complex avatar to be isolated from the plurality of segments represented by a tree of joints. (An example tree of joints for a humanoid skeleton, such as shown inFIG.3is illustrated inFIG.5A.) A segment can be selected by identifying locations of joints at terminal ends of the segment to be isolated onto portions of the rendered version of the complex avatar. The simplified avatars service146can be configured to methodically progress through the tree of joints one at a time for a humanoid avatar.

Before the selected segment of the complex avatar can be isolated, simplified avatars service146first reparents the tree of joints to make one of the joints at the terminal ends of the segment a new root node of the tree of joints. Typically, the joint that will become the new root node is at one of the terminal ends of the segment that already occupies the higher level in the tree of joints.

The reparenting the tree of joints comprises iteratively reorganizing the tree of joints to promote the desired node in the tree of joints one level until the desired node is at the top of the tree of joints. The relative organization of the tree of joints remains the same, whereby each node in the tree of joints remains connected to a same neighbor in the tree of joints as the tree of joints is iteratively reorganized. It is only the level in the tree of joints that changes for the respective joints. If this process were animated, it might appear as if the terminal ends of the selected segment remain in place while the rest of the tree of joints rotates about the selected segment until one of the terminal ends of the selected segment is at the top of the tree of joints.

FIG.5A,FIG.5Billustrate the reparenting of the tree of joints502andFIG.6A,FIG.6B, andFIG.6C, illustrate the isolation of the selected isolated segment. As addressed below,FIG.5AandFIG.5Billustrates reparenting the tree of joints so that the right shoulder has become the root node of the tree of joints, thereby allowing the right upper arm (between the right shoulder and the right elbow) to be the isolated segment.

Once the tree of joints has been reparented to bring the segment to be isolated to the root of the tree of joints, the isolating of the segments of the complex avatar can be completed by scaling the terminal ends of the selected segment to less than 1% scale. In some embodiments, the less than 1% scale is 1/10,000th to 1/100,000th scale. In some embodiments, the less than 1% is greater than zero to avoid potential errors from the rendering engine.

The scaling of the terminal ends (joints at either end) of the isolated segment results in a collapsing of portions of the rendered version of the complex avatar other than the isolated segment into the respective joints at the terminal ends of the isolated segment. This scaling results in a rendering where only the isolated segment of the complex avatar is visible. The rest of the complex avatar is still present, but since everything from the terminal end of the selected segment to be isolated to the bottom of the tree has been scaled to a very small scale, it is no longer visible. This is illustrated inFIG.6B.

Scaling the terminal ends of the selected segment to be isolated has the advantage of rendering the isolated segment of the complex avatar as a closed volume. It may be possible to isolate the segment by other methods. For example, it might be possible to mask the rest of the complex avatar, crop the selected segment of the avatar, or cause the rest of the avatar not to be rendered. However, these techniques would likely cause the isolated segment to appear hollow when viewed from certain angles, and this would have the undesirable consequence of making the simplified version of the complex avatar look like a collection of disconnected parts when viewed from certain angles. However, it should be understood that the present technology does encompass these other techniques as the open volume consequence of these methods can be addressed by other known techniques in the art.

According to some embodiments, the method includes capturing images of the isolated segments of the complex avatar from a variety of directions using a virtual camera at block408. For example, the simplified avatars service146may capture images of the isolated segments of the complex avatar from a variety of directions using a virtual camera. For each direction from which the images of the isolated segment were captured, a visual appearance and a shape of the segment of the rendered complex avatar is captured. In some embodiments, the images of the segments are associated with a vector from the isolated segments to the virtual camera used to capture the image.

In some embodiments, images are typically captured from between 50-100 orientations, but the images can be captured from any number of orientations. In some embodiments, the different orientations are obtained by moving the isolated segment of the complex avatar into different orientations about the virtual camera. Alternatively, the virtual camera can move about the isolated segment of the complex avatar. Alternatively still, another method of obtaining the captured images from different orientations includes establishing a plurality of the virtual cameras at different positions about the isolated segment of the complex avatar to capture different orientations of the isolated segment.

The captured images are simple sprites and do not include a high number of polygons, complicated mesh patterns, custom animations, metadata, scripts, or shader properties. In some embodiments, even though the captured images were taken of a 3-D object they do not provide depth information.

According to some embodiments, the method includes sampling a depth buffer of points in 3-D space for locations of pixels making up each of the isolated segments of the complex avatar at block410. For example, the simplified avatars service146may sample a depth buffer of points in 3-D space for locations of pixels making up each of the isolated segments of the complex avatar. The information captured from the depth buffer provides information about the 3-D shape of the complex avatar.

In some embodiments, the sampling of the depth buffer is from the same point of view as the virtual camera when capturing the images of the isolated segment of the complex avatar. In such embodiments, the samples from the depth buffer can be associated with the captured images taken from the virtual camera from the same direction, or both sets of information can be associated with the same vector.

In some embodiments, the samples from the depth buffer and the images of the isolated segments can be used to reproduce the isolated segment in a simplified version. In one method, an isolated segment and depth buffer matching a point of view can be selected and the captured image of the isolated segment can be mapped onto a volume formed using the samples from the depth buffer and this will result in providing a reasonable approximation of the appearance of the isolated segment from that point of view. In this method only the visible portion of the isolated segment is generated. In another method, the collection of samples from the depth buffer can be used to generate a complete 3-D depth map of the isolated segment, and the captured image from a particular point of view can be mapped onto the complete 3-D depth map. In this method, a full 3-D volume of the isolated segment is created as an invisible mesh and the image of the isolated segment is placed on top of the full 3-D volume such that while the isolated segment occupies a 3-D volume, only a portion of it is visible at any time. In both methods, the only portion that would be visible corresponds to the point of view so that the invisible portions of the isolated segment are not noticeable from that point of view.

According to some embodiments, the method includes storing the captured images for each of the isolated segments of the complex avatar along with the vector from the segment to the virtual camera used to capture the image at block412. For example, the simplified avatars service146may store the captured images and the depth information for each of the isolated segments of the complex avatar along with the vector from the segment to the virtual camera used to capture the image. Collectively the captured images for each of the isolated segments and the depth information and the vectors toward the virtual camera for each of the captured images makes up the avatar assets for the simplified version of the complex avatar.

Once a simplified version of a complex avatar exists, it does not mean that the simplified version will always be served to requesting clients. When the requesting client is able to render the complex avatar, this set of avatar assets will be served. Accordingly, the avatars API136must determine which avatar version to serve to requesting clients. In some instances, one client will render the complex avatar in its instance of a virtual world while another client will render the simplified version of the avatar for its instance of the virtual world.

In some embodiments, a requesting client might explicitly request a simplified version of the avatar, and in that instance, the avatars API136can serve the simplified version of the avatar.

According to some embodiments, the method includes determining that a client requesting assets making up the avatar should receive the simplified version of the complex avatar at block416. For example, the avatars API136may determine that a client requesting assets making up the avatar should receive the simplified version of the complex avatar. In some embodiments, the client will use the collection of images of the segments of the complex avatar and a collection of depth buffer points for the segments of the complex avatar to render the simplified version of the avatar.

According to some embodiments, the method includes determining that a client requesting assets making up the complex avatar can support a high-resolution version of the complex avatar at block418. For example, the avatars API136may determine that a client requesting assets making up the complex avatar can support a high-resolution version of the complex avatar. In some embodiments, the client can render some of the complex avatars in the virtual world using the simplified version of the complex avatar.

In some embodiments, generating the avatar assets for the simplified version of the complex avatar is performed in an offline or in a non-time sensitive process. In other words, the generating of the avatar is not performed at the time a simplified version of the avatar is requested. Rather the generating of the simplified version of the complex avatar can be performed when the avatar is first stored in avatar assets databases138. Alternatively, a workflow to create the simplified version of the avatar can be queued once a client reports unsuitable performance in attempting to render the complex avatar. In some embodiments, a user can select their own avatar, or an avatar in a virtual world, and can request to have a simplified version of the avatar created.

FIG.4Billustrates an example method for creating a simplified version of a complex avatar in alternate poses. As described with respect toFIG.4A, the complex avatar is rendered in a t-pose and then segments are isolated and images are captured. However, the result of capturing the images in just one pose is that some common poses for the avatar might not be able to be rendered. For example, the hands of the avatar might be able to open or close or point, and the mouth of an avatar likely animates open and closed or from similes to frowns and during speech. These poses are not the result of articulations in the joints from the tree of joints so they would not be supported by the many different camera angles addressed above. Accordingly, to support these common poses that would not be supported by the steps disclosed with respect toFIG.4A, it may be required to pose the avatar, or at least portions of the avatar in additional poses.

According to some embodiments, the method includes rendering the complex avatar in a second pose at block420. For example, the simplified avatars service146may render the complex avatar in a second pose.

According to some embodiments, the method includes capturing images of the isolated segments of the complex avatar in the second pose at block422. For example, the simplified avatars service146may capture the images of the isolated segments of the complex avatar in the second pose. For example, the second pose might include open hands, closed hands, and different mouth states, etc. While the present description refers a second pose, it will be appreciated by those of ordinary skill in the art that this is merely to distinguish the second pose from the first pose, and that the complex avatar, or portions thereof, can be rendered in any number of poses and the method can include capturing images of the complex avatar in any number of poses.

FIG.5AandFIG.5Billustrate the reparenting of the tree of joints502. InFIG.5A, the tree of joints502illustrates the root of the tree of joints as being between the lower back and the hips.

Each of those branches has branches coming off them that represent the attached limbs such as shoulders branching off the chest and lower legs branching off the upper legs. Branches that are near the root of the hierarchy are above or higher in the tree of joints than those that are further from the root.

As addressed above, the present technology isolates each limb segment of the avatar (e.g., the upper arm from shoulder to elbow) using functionality provided by gaming engines, such as the gaming engine from UNITY SOFTWARE INC., which help determine which locations represent the main joints of a humanoid.

Also as addressed above with respect to block406, part of isolating a segment of the humanoid, involves reparenting the tree of joints. The present technology can reparent the tree of joints502for every segment to be isolated.

FIG.5Billustrates an example of the reparenting of the tree of joints to move the joint for the right shoulder504to be the parent node, or highest node, or top node of the tree such that every other joint is subordinate to the right shoulder504in the tree of joints502. It is an iterative process to arrive at the tree of joints502having the right shoulder504being the parent node. The present technology eventually results in a new hierarchy of every joint above the isolated segment. For example inFIG.5Bthe right shoulder504is now the root of the hierarchy, and the joint that used to be above the isolated segment (such as the neck or chest) is moved to the previous position of the right shoulder504.

FIG.6A,FIG.6B, andFIG.6Cillustrate the general steps addressed above regarding rendering the complex avatar in block404, isolating segments of the complex avatar in block406, and capturing images of the complex avatar in block408.

FIG.6Aillustrates the rendered version of the complex avatar602. Although it is difficult to illustrate in line drawing form, the rendered version of the complex avatar602can be highly detailed and can have a complex 3-D shape or require a lot of triangles to be drawn to render properly. The complexity or detail can cause the avatar to be difficult to render in some environments.

FIG.6Billustrates an isolated segment. In this example the isolated segment is the right upper arm.FIG.6Billustrates the same version of the upper right arm as rendered inFIG.6A, except that the rest of the complex avatar is not visible. In some embodiments, the rest of the complex avatar has been scaled to such a small size that it is not visible.

FIG.6Cillustrates the isolated segment as a captured image606. Although difficult to illustrate in line drawing form, the captured image is of a smaller file size and easier to render. In some embodiments, the captured image may be of somewhat lower quality or resolution that the complex version of the isolated segment, and might not include any associated scripts or animations. The isolated segment as the captured image along with the depth buffer data can be used to build a copy of the isolated segment as viewed from the direction of the camera used to capture it.

In the present technology, instead of drawing the original character model, the present technology selects the best fitting pre-generated image and shape for each limb segment, which is the one that was captured using a camera that was facing the same way as the current view camera, or the closest one to that direction, and draw it instead. This reduces the size of the character assets such that more of them can be loaded into memory on a given device and it reduces the geometric complexity of the character such that it takes less resources to render it to the screen. The size of the output using the current technology does not grow with the size of the data of the input complex avatar. A normally proportioned humanoid simplified version of an avatar can export to a file that is about 3-4 MB even if the original complex avatar was 100 MB or more.

FIG.7illustrates an example method for rendering a simplified version of the complex avatar from the captured images and depth buffer data. Although the example method depicts a particular sequence of operations, the sequence may be altered without departing from the scope of the present disclosure. For example, some of the operations depicted may be performed in parallel or in a different sequence that does not materially affect the function of the method. In other examples, different components of an example device or system that implements the method may perform functions at substantially the same time or in a specific sequence.

According to some embodiments, the method includes downloading avatar assets for a simplified version of a complex avatar at block702. For example, client104can request avatar assets for an avatar from avatars API136. Avatars API136can determine that client104should be served a simplified version of the avatar assets, and client104can download the avatar assets for a simplified version of a complex avatar. The avatar assets can be for the local avatar118or remote avatar126. Generally, the avatar assets will be for the remote avatar126when the local player116is controlling their avatar from a first person point of view. But in instances where the local player116is controlling their avatar from a perspective view the avatar assets can pertain to the local avatar118

According to some embodiments, the method includes determining a current pose of the avatar and a point of view from which to render the avatar with respect to the avatar at block704. For example, the client104may determine a current pose of the avatar and a point of view with respect to the avatar. Generally, the point of view is from the perspective of the local player116viewing a remote avatar126. But in some embodiments, the point of view could be from a perspective view camera and the avatar could be the local avatar118or the remote avatar126.

According to some embodiments, the method includes selecting a captured image of an isolated segment from a collection of avatar assets at block706. For example, the client104may select a captured image of an isolated segment from a collection of avatar assets. In some embodiments, the selected captured image is associated with the vector from the isolated segment to a virtual camera used to capture the captured image that most closely matches a vector from the isolated segment to the point of view.

According to some embodiments, the method includes generating a square mesh representing an area of at least a portion of the complex avatar corresponding to the isolated segment at block708. For example, the client104may generate a square mesh representing an area of at least a portion of the complex avatar corresponding to the isolated segment. While the term square mesh is used, any shape of mesh can be used.

According to some embodiments, the method includes locating the square mesh on top of a skeleton outlined from a tree of joints making up a humanoid avatar at block710. For example, the client104can determine where joints of a skeleton in a tree of joints should be located in the virtual world and can locate the square mesh on top of a segment of the skeleton corresponding to the isolated segment selected as addressed with respect to block706.

The purpose of the square mesh is to define the 3-D shape of the avatar or segment of the avatar. However, the square mesh is simply a 2-D area and therefore needs further processing to yield a shape that is similar to the shape of the isolated segment of the complex avatar. According to some embodiments, the method includes deforming the shape of the square mesh using values from a depth buffer to approximately represent the shape of the isolated segment of the complex avatar at block712. For example, the client104may deform the shape of the square mesh using values from a depth buffer to approximately represent the shape of the isolated segment of the complex avatar. In particular, the values from the depth buffer can correspond to x, y vertices of the square mesh that need to be adjusted in the z-plane to deform the 2-D mesh into a 3-D shape.

According to some embodiments, the method includes mapping the selected captured image of the isolated segment of the avatar to the square mesh after deforming the shape of the square mesh at block714. For example, the client104may map the selected captured image of the isolated segment of the avatar to the square mesh.

According to some embodiments, the method includes scaling the selected captured image of the isolated segment based on the distance from point of view at block716. For example, the client104may scale the selected captured image of the isolated segment based on the distance from point of view.

As will be appreciated by those of ordinary skill in the art the functions described with respect to block706through block716need to be repeated for each isolated segment making up the simplified version of the complex avatar. The functions described with respect to block706through block716can be performed in parallel for each segment or can be performed in series.

According to some embodiments, the method includes re-rendering the simplified version of the avatar every frame of a game that includes the avatar at block718. For example, the client104may re-render the simplified version of the avatar every frame of a game that includes the avatar.

FIG.8illustrates an example method for supporting non-humanoid complex avatars. Although the example method depicts a particular sequence of operations, the sequence may be altered without departing from the scope of the present disclosure. For example, some of the operations depicted may be performed in parallel or in a different sequence that does not materially affect the function of the method. In other examples, different components of an example device or system that implements the method may perform functions at substantially the same time or in a specific sequence.

According to some embodiments, the method includes determining that the complex avatar is non-humanoid at block802. For example, the simplified avatars service146illustrated may determine that the complex avatar is non-humanoid if it cannot map the avatar to a humanoid tree of joints.

According to some embodiments, one method of handling a non-humanoid avatar is to treat the entire complex avatar as a single segment at block804. For example, the simplified avatars service146may treat the entire complex avatar as a single segment. Accordingly, the entire avatar could be imaged in a single pose or multiple poses as addressed with respect to the humanoid avatar inFIG.4A. Rather than isolating multiple segments, the whole avatar can be treated as a single avatar and images and depth buffer data can be captured from multiple angles.

According to some embodiments, another method of handling non-humanoid complex avatars is to play through all animations for which the non-humanoid complex avatar is capable at block806. For example, the simplified avatars service146may play through all animations for which the non-humanoid complex avatar is capable.

According to some embodiments, the method includes defining a custom tree of joints for the non-humanoid complex avatar at block340. For example, the simplified avatars service146may define a custom tree of joints for the non-humanoid complex avatar. By observing all movements associated with the non-humanoid complex avatar, the simplified avatars service146can identify all points of articulation and create a tree of joints for the non-humanoid complex avatar.

Once a tree of joints has been defined, the simplified avatars service146can create avatar assets for the non-humanoid complex avatar in the same way as it would for a humanoid avatar described herein.

Accordingly, one advantageous aspect of the present technology is that it is capable of creating a simplified version of any complex avatar or any user generated object. Thus, users can continue to generate highly complex works that will render as intended in environments that are capable, and will be able to render in a simplified form in environments that are less capable.

FIG.9shows an example of computing system900, which can be for example any computing device making up client device106, or web services110, or any component thereof in which the components of the system are in communication with each other using connection902. Connection902can be a physical connection via a bus, or a direct connection into processor904, such as in a chipset architecture. Connection902can also be a virtual connection, networked connection, or logical connection.

Example computing system900includes at least one processing unit (CPU or processor)904and connection902that couples various system components including system memory908, such as read-only memory (ROM)910and random access memory (RAM)912to processor904. Computing system900can include a cache of high-speed memory906connected directly with, in close proximity to, or integrated as part of processor904.

Processor904can include any general purpose processor and a hardware service or software service, such as services916,918, and920stored in storage device914, configured to control processor904as well as a special-purpose processor where software instructions are incorporated into the actual processor design. Processor904may essentially be a completely self-contained computing system, containing multiple cores or processors, a bus, memory controller, cache, etc. A multi-core processor may be symmetric or asymmetric.

To enable user interaction, computing system900includes an input device926, which can represent any number of input mechanisms, such as a microphone for speech, a touch-sensitive screen for gesture or graphical input, keyboard, mouse, motion input, speech, etc. Computing system900can also include output device922, which can be one or more of a number of output mechanisms known to those of skill in the art. In some instances, multimodal systems can enable a user to provide multiple types of input/output to communicate with computing system900. Computing system900can include communication interface924, which can generally govern and manage the user input and system output. There is no restriction on operating on any particular hardware arrangement, and therefore the basic features here may easily be substituted for improved hardware or firmware arrangements as they are developed.

The storage device914can include software services, servers, services, etc., that when the code that defines such software is executed by the processor904, it causes the system to perform a function. In some embodiments, a hardware service that performs a particular function can include the software component stored in a computer-readable medium in connection with the necessary hardware components, such as processor904, connection902, output device922, etc., to carry out the function.

Aspect 1. A method for generating a simplified version of a complex avatar made up of a plurality of segments, the method comprising: isolating segments of the complex avatar from the plurality of segments of a rendered version of the complex avatar; for each of the isolated segments, capturing images of the isolated segments of the complex avatar from a variety of directions using a virtual camera, wherein the images of the segments are associated with a vector from the isolated segments to the virtual camera used to capture the image; and storing the captured images for each of the isolated segments of the complex avatar along with the vector from the segment to the virtual camera used to capture the image, collectively the captured images make up simplified avatar assets used to render a simplified version of the complex avatar.

Aspect 2. The method of Aspect 1, further comprising: sampling a depth buffer of points in 3-D space for locations of pixels making up each of the isolated segments of the complex avatar.

Aspect 3. The method of any of Aspects 1 to 2, further comprising: storing the points on the isolated segment sampled from the depth buffer as part of the simplified avatar assets.

Aspect 4. The method of any of Aspects 1 to 3, wherein the sampling the depth buffer of points for each of the isolated segments further comprises: sampling the depth buffer from a point of view of the virtual camera when capturing the images of the isolated segment of the complex avatar.

Aspect 5. The method of any of Aspects 1 to 4, whereby for each direction from which the images of the isolated segment were captured, a visual appearance and a shape of the segment of the rendered complex avatar is captured.

Aspect 6. The method of any of Aspects 1 to 5, wherein the capturing the images of the isolated segment of the complex avatar from a variety of directions further comprises: moving the isolated segment of the complex avatar into different orientations about the virtual camera.

Aspect 7. The method of any of Aspects 1 to 6, wherein images are captured from between 50-100 orientations, but can be any number of orientations.

Aspect 8. The method of any of Aspects 1 to 7, wherein the capturing the images of the isolated segment of the complex avatar from a variety of angles further comprises: moving the virtual camera about the isolated segment of the complex avatar to capture different orientations of the isolated segment.

Aspect 9. The method of any of Aspects 1 to 8, wherein the capturing the images of the isolated segment of the complex avatar from a variety of angles further comprises: establishing a plurality of the virtual cameras at different positions about the isolated segment of the complex avatar to capture different orientations of the isolated segment.

Aspect 10. The method of any of Aspects 1 to 9, wherein the rendered version of the complex avatar is rendered in a t-pose.

Aspect 11. The method of any of Aspects 1 to 10, wherein the complex avatar is a humanoid complex avatar defined by a tree of joints, wherein a segment of the complex avatar is a portion of the complex avatar between two joints in the tree of joints.

Aspect 12. The method of any of Aspects 1 to 11, wherein the isolating the segment of the complex avatar from the plurality of segments of a rendered version of the complex avatar further comprises: selecting the segment of the complex avatar to be isolated from the plurality of segments represented by the tree of joints; identify locations of joints at terminal ends of the segment to be isolated onto portions of the rendered version of the complex avatar; reparenting the tree of joints to make one of the joints at the terminal ends of the segment a new root node of the tree of joints; scaling the portions of the rendered complex avatar where the joints at terminal ends of the segment to be isolated are located to less than 1% scale, thereby collapsing portions of the rendered version of the complex avatar other than the isolated segment into the respective joints at the terminal ends of the isolated segment resulting in a rendering of the isolated segment of the complex avatar.

Aspect 13. The method of any of Aspects 1 to 12, wherein the less than 1% scale is 1/10,000th to 1/100,000th scale.

Aspect 14. The method of any of Aspects 1 to 13, whereby the result of the scaling is that the isolated segment of the complex avatar has a closed volume.

Aspect 15. The method of any of Aspects 1 to 14, wherein the reparenting the tree of joints further comprises: whereby each node in the tree of joints remains connected a same neighbor in the tree of joints as the tree of joints is iteratively reorganized.

Aspect 16. The method of any of Aspects 1 to 15, further comprising: rendering the complex avatar in another pose; and recapturing the images of the isolated segments of the complex avatar in the another pose, for example the another pose can be open hands, closed hands, mouth, etc.

Aspect 17. The method of any of Aspects 1 to 16, the method further comprising: storing the avatar assets making up the complex avatar; and storing avatar assets making up the simplified version of the complex avatar as a collection of images of the isolated segments of the complex avatar and a collection of depth buffer points for the isolated segments of the complex avatar in an avatar assets database.

Aspect 18. The method of any of Aspects 1 to 17, further comprising: determining that a client requesting assets making up the avatar should receive the simplified version of the complex avatar; and whereby the client will use the collection of images of the segments of the complex avatar and a collection of depth buffer points for the segments of the complex avatar to render the simplified version of the avatar.

Aspect 19. The method of any of Aspects 1 to 18, wherein generating the avatar assets for the simplified version of the complex avatar, is performed offline using cloud computing resources.

Aspect 20. The method of any of Aspects 1 to 19, further comprising: determining that a client requesting assets making up the complex avatar can support a high-resolution version of the complex avatar; and whereby the client can render some of the complex avatars in the virtual world using the simplified version of the complex avatar.

Aspect 21. The method of any of Aspects 1 to 20, further comprising: determining that the complex avatar does not conform to a humanoid tree of joints; and treating the entire complex avatar as a single segment.

Aspect 22. The method of any of Aspects 1 to 21, further comprising: determining that the complex avatar is non-humanoid, whereby it does not conform to a humanoid tree of joints; playing through all animations for which the non-humanoid complex avatar is capable; and defining a custom tree of joints for the non-humanoid complex avatar.

Aspect 23. The method of any of Aspects 1 to 22, wherein the complex avatar is made up of a high number of polygons, complicated mesh patterns, custom animations, metadata, scripts, or shader properties, etc.

Aspect 24. The method of any of Aspects 1 to 23, wherein the captured images do not include a high number of polygons, complicated mesh patterns, custom animations, metadata, scripts, or shader properties.

Aspect 25. The method of any of Aspects 1 to 24, further comprising: rendering the simplified version of the complex avatar from the captured images of the segment of the complex avatar.

Aspect 26. A method for rendering a simplified version of an avatar, where an original version of the avatar is a complex avatar, the method comprising: determining a current pose of the avatar and a point of view from the perspective of the local player with respect to the avatar; selecting a captured image of an isolated segment from a collection of avatar assets, wherein the selected captured image is associated with the vector from the isolated segment to a virtual camera used to capture the captured image that most closely matches the point of view; generating a square mesh representing a volume of at least a portion of the complex avatar corresponding to the isolated segment; locating the square mesh on top of a skeleton outlined from a tree of joints making up a humanoid avatar; deforming the shape of the square mesh using values from a depth buffer to approximately represent the shape of the isolated segment of the complex avatar; mapping the selected captured image of the isolated segment of the avatar to the square mesh after deforming the shape of the square mesh; and scaling the selected captured image of the isolated segment based on the distance from point of view.

Aspect 27. The method of Aspect 26 further comprising: recursively performing the rendering of the simplified version of the avatar from the collection of avatar assets for each isolated segment of the avatar to render the simplified version of the avatar.

Aspect 28. The method of any of Aspects 26 to 27 further comprising: re-rendering the simplified version of the avatar every frame of a game that includes the avatar.