Textured mesh building

Systems and methods are provided for receiving a two-dimensional (2D) image comprising a 2D object; identifying a contour of the 2D object; generating a three-dimensional (3D) mesh based on the contour of the 2D object; and applying a texture of the 2D object to the 3D mesh to output a 3D object representing the 2D object.

TECHNICAL FIELD

The present disclosure relates generally to visual presentations of virtual content and, more particularly, to rendering a three-dimensional (3D) object within a real-world environment captured in a camera feed of a computing device.

BACKGROUND

Virtual rendering systems can be used to create engaging and entertaining augmented reality experiences, in which 3D virtual object graphics content appear to be present in the real-world. Such systems allow a user to select from a predefined list of 3D objects and display the selected 3D objects in the view of a camera feed.

DETAILED DESCRIPTION

Among other things, embodiments of the present disclosure improve the functionality of electronic messaging and imaging software and systems by providing functionality that allows users to create virtual 3D objects from a two-dimensional (2D) image, such as a caption, user generated content, pre-generated content, curated content, geofilter, picture, sticker, emoji, and a GIG. The virtual 3D objects that are created are then rendered, as if they exist in real-world environments. For example, media overlays of 3D images can be generated by the system and displayed in conjunction with real-world environment content (e.g., images and/or video) generated by an image-capturing device (e.g., a digital camera).

Users are always seeking new ways to interact with content as if it exists in the real world. Typical systems enable a user to insert a 2D image into a video frame and control its position in two dimensions (e.g., x and y coordinates). However, simply inserting a 2D image into a video frame without considering other objects (e.g., surfaces) in the video frame makes the resulting video frame with the 2D image look un-realistic, particularly because depth of the object cannot be utilized. In addition, because the 2D image that can be inserted lacks any 3D qualities, the typical systems are incapable of considering how to place and position the 2D image relative to real objects that appear in the video frame. Some systems allow a user to choose from a predefined list of 3D objects to insert into a real-word environment that is depicted in a camera feed. While such systems generally work well in presenting such 3D objects in the real-world environment, the lack of ability to customize and manipulate the 3D content by a user makes the systems less appealing and interesting to users.

The embodiments of the present disclosure allow a user to create a virtual 3D object from any 2D image. This enables the user to add the 3D virtual object that has been created into a real-world video frame that contains real-world objects. The user can control the position of the object not only in two dimensions but in three dimensions (e.g., x, y, and z coordinates). Because the 2D image has been converted to a 3D object, the disclosed embodiments are able to track and reposition or re-align the 3D object in real time relative to real 3D objects that appear in the scene. Particularly, the generated 3D object has real 3D properties (e.g., depth, width, height, and length) which can be used to adjust a position of the 3D object relative to the 3D properties of objects in a real-world scene. For example, the 3D object can be placed on top of a table that appears in the scene and can be repositioned relative to a surface of the table. As the camera moves around, the system can continue tracking the 3D object relative to the table surface which makes the overall image containing real and virtual objects appear more realistic. In this way, the embodiments of the present disclosure allow a user to generate and modify frames of a video by adding 3D virtual objects and to interact with those objects in new ways.

FIG. 1is a block diagram showing an example messaging system100for exchanging data (e.g., messages and associated content) over a network. The messaging system100includes multiple client devices102, each of which hosts a number of applications including a messaging client application104. Each messaging client application104is communicatively coupled to other instances of the messaging client application104and a messaging server system108via a network106(e.g., the Internet).

Accordingly, each messaging client application104is able to communicate and exchange data with another messaging client application104and with the messaging server system108via the network106. The data exchanged between messaging client applications104, and between a messaging client application104and the messaging server system108, includes functions (e.g., commands to invoke functions) as well as payload data (e.g., text, audio, video, or other multimedia data).

In some embodiments, the messaging client application104presents a graphical user interface (GUI) to a user for selecting or creating a given 2D image of an object. For example, the user can type text in 2D and input that 2D text as the selected 2D object image. The messaging client application104processes the 2D image of the object to generate a virtual 3D object from the 2D image. A user can activate a camera of the messaging client application104to view images of the user's real-world surroundings (e.g., the camera feed) in real-time. The user can instruct the messaging client application104to add the newly created virtual 3D object to the real-world images being captured by the camera. In this way, the user can add a virtual 3D object to real-world objects depicted in the camera feed. The user can manipulate the virtual 3D object to reposition the virtual object relative to the real-world objects. In some embodiments, the user can capture and store a video or image that includes the virtual 3D object and the real-world objects and share the video or image with another user of another messaging client application104.

The messaging server system108provides server-side functionality via the network106to a particular messaging client application104. While certain functions of the messaging system100are described herein as being performed by either a messaging client application104or by the messaging server system108, it will be appreciated that the location of certain functionality either within the messaging client application104or the messaging server system108is a design choice. For example, it may be technically preferable to initially deploy certain technology and functionality within the messaging server system108, but to later migrate this technology and functionality to the messaging client application104where a client device102has a sufficient processing capacity.

Dealing specifically with the API server110, this server receives and transmits message data (e.g., commands and message payloads) between the client device102and the application server112. Specifically, the API server110provides a set of interfaces (e.g., routines and protocols) that can be called or queried by the messaging client application104in order to invoke functionality of the application server112. The API server110exposes various functions supported by the application server112, including account registration, login functionality, the sending of messages, via the application server112, from a particular messaging client application104to another messaging client application104, the sending of media files (e.g., images or video) from a messaging client application104to the messaging server application114, and for possible access by another messaging client application104, the setting of a collection of media data (e.g., story), the retrieval of such collections, the retrieval of a list of friends of a user of a client device102, the retrieval of messages and content, the adding and deleting of friends to a social graph, the location of friends within a social graph, opening an application event (e.g., relating to the messaging client application104).

The application server112also includes an image processing system116that is dedicated to performing various image processing operations, typically with respect to images or video received within the payload of a message at the messaging server application114.

The social network system122supports various social networking functions and services and makes these functions and services available to the messaging server application114. To this end, the social network system122maintains and accesses an entity graph within the database120. Examples of functions and services supported by the social network system122include the identification of other users of the messaging system100with which a particular user has relationships or is “following” and also the identification of other entities and interests of a particular user.

The application server112is communicatively coupled to a database server118, which facilitates access to a database120in which is stored data associated with messages processed by the messaging server application114.

FIG. 2is block diagram illustrating further details regarding the messaging system100, according to example embodiments. Specifically, the messaging system100is shown to comprise the messaging client application104and the application server112, which in turn embody a number of some subsystems, namely an ephemeral timer system202, a collection management system204, and an annotation system206.

The ephemeral timer system202is responsible for enforcing the temporary access to content permitted by the messaging client application104and the messaging server application114. To this end, the ephemeral timer system202incorporates a number of timers that, based on duration and display parameters associated with a message, or collection of messages (e.g., a story), selectively display and enable access to messages and associated content via the messaging client application104. Further details regarding the operation of the ephemeral timer system202are provided below.

The collection management system204furthermore includes a curation interface208that allows a collection manager to manage and curate a particular collection of content. For example, the curation interface208enables an event organizer to curate a collection of content relating to a specific event delete inappropriate content or redundant messages). Additionally, the collection management system204employs machine vision (or image recognition technology) and content rules to automatically curate a content collection. In certain embodiments, compensation may be paid to a user for inclusion of user generated content into a collection. In such cases, the curation interface208operates to automatically make payments to such users for the use of their content.

The annotation system206provides various functions that enable a user to annotate or otherwise modify or edit media content associated with a message. For example, the annotation system206provides functions related to the generation and publishing of media overlays for messages processed by the messaging system100. The annotation system206operatively supplies a media overlay (e.g., a filter or lens) to the messaging client application104. In another example, the annotation system206operatively supplies a media overlay to the messaging client application104based on other information, such as social network information of the user of the client device102. A media overlay may include audio and visual content and visual effects. Examples of audio and visual content include pictures, texts, logos, animations, and sound effects. An example of a visual effect includes color overlaying.

The audio and visual content or the visual effects can be applied to a media content item (e.g., a photo) at the client device102. For example, the media overlay including text that can be overlaid on top of an image or video generated by the client device102. In another example, the media overlay includes an identification of a location overlay (e.g., Venice beach), a name of a live event, or a name of a merchant overlay (e.g., Beach Coffee House).

The annotation system206includes a 3D object generation system210that provides functionality to receive a 2D object and generate virtual 3D objects, from the 2D object, and display and track the virtual 3D object, at positions relative to the client device102, within a 3D space captured within a camera feed of the client device102(also referred to by those of ordinary skill in the art as a “camera stream,” “a video stream,” or a “video feed”). The virtual 3D objects generated, displayed, and tracked by the 3D object generation system210include 3D objects generated from a 2D object. A 3D object represents any 2D provided user generated content, curated content, images, text, video, animation or other visual item selected by a user or automatically identified. In an embodiment, the 2D object and the 3D object are simultaneously presented on the display. In such circumstances, the 3D object moves with and relative to the video feed in which the 3D object is positioned by the user while the 2D object remains stationary at a predetermined or user selected position on the display. In an embodiment, the 2D object includes a 2D video or animation. In such circumstances, the 2D video or animation is used to generate a 3D object that includes a 3D animation or 3D video which loops or cycles in the video feed in which the 3D object is positioned (e.g., a virtual dancing hot dog).

The 3D object generation system210provides functionality to enable users to author, edit, and preview 3D objects by supplying a 2D object. To this end, the 3D object generation system210includes an editing interface212and a preview interface214. The editing interface212allows a user to author and select a 2D object (e.g., the user can select a 2D image or video from a list of images and videos or can manually type in a set of characters corresponding to text). The editing interface212enables users to edit the 2D object using keyboard input and other types of input including touch screen based gestures. For example, the user can change the scale, color scheme, size, or any other visual attribute of the 2D content. After the user is satisfied with the edited 2D object, the user can instruct the 3D object generation system210to create a virtual 3D object from the 2D object. The preview interface214allows a user to preview and review the generated 3D object before generating a message that includes the 3D object. The preview interface214may also enable the user to edit the presentation of the 3D objects (e.g., by changing a scale, orientation, or placement of the 3D object on the display screen). The 3D object generation system210creates the 3D object from the 2D object in accordance with the process described in connection withFIGS. 7 and 8.

The 3D object generation system210may cause a 3D object to be displayed (e.g., on a display of the client device102) at positions in a 3D space captured within the camera feed based on a reference surface (e.g., the ground) detected in the 3D space. As will be discussed in further detail below, the 3D object generation system210comprises a redundant tracking system comprising a set of tracking subsystems configured to track a 3D object at a position in 3D space based on a set of tracking indicia and transition between tracking subsystems. The 3D object generation system210may further transition between tracking with six degrees of freedom (6DoF) and tracking with three degrees of freedom (3DoF) based on an availability of the tracking indicia. The 3D object generation system210, in an embodiment, tracks and displays the 3D object at a position in 3D space captured within the camera feed while simultaneously presenting the corresponding 2D object in a static location Namely, the 2D object is not tracked in 3D space as the camera changes positions in 3D space altering the camera feed in which the 3D object is presented.

FIG. 3is a schematic diagram300illustrating data, which may be stored in the database120of the messaging server system108, according to certain example embodiments. While the content of the database120is shown to comprise a number of tables, it will be appreciated that the data could be stored in other types of data structures (e.g., as an object-oriented database).

The database120includes message data stored within a message table314. An entity table302stores entity data, including an entity graph304. Entities for which records are maintained within the entity table302may include individuals, corporate entities, organizations, 2D and/or 3D objects, 3D object templates, 3D object textures, places, events, and so forth. Regardless of type, any entity regarding which the messaging server system108stores data may be a recognized entity. Each entity is provided with a unique identifier, as well as an entity type identifier (not shown).

The entity graph304furthermore stores information regarding relationships and associations between entities. Such relationships may be social, professional (e.g., work at a common corporation or organization), interested-based, or activity-based, merely for example.

The database120also stores annotation data, in the example form of filters and lenses, in an annotation table312. Filters and lenses for which data is stored within the annotation table312are associated with and applied to videos (for which data is stored in a video table310) and/or images (for which data is stored in an image table308). Filters are overlays that are displayed as overlaid on an image or video during presentation to a recipient user. Lenses, on the other hand, include real-time special effect and/or sounds that may be added to images of a camera feed while a user is composing a message. In comparison, filters are applied to an image or video after the image or video is captured at the client device102while a lens is applied to the camera feed of the client device102such that when an image or video is captured at the client device102with a lens applied, the applied lens is incorporated as part of the image or video that is generated. Filters and lenses may be of various types, including user-selected filters and lenses from a gallery of filters or a gallery of lenses presented to a sending user by the messaging client application104when the sending user is composing a message.

As mentioned above, the video table310stores video data which, in one embodiment, is associated with messages for which records are maintained within the message table314. Similarly, the image table308stores image data associated with messages for which message data is stored in the entity table302. The entity table302may associate various annotations from the annotation table312with various images and videos stored in the image table308and the video table310.

FIG. 4is a schematic diagram illustrating a structure of a message400, according to some some embodiments, generated by a messaging client application104for communication to a further messaging client application104or the messaging server application114. The content of a particular message400is used to populate the message table314stored within the database120, accessible by the messaging server application114. Similarly, the content of a message400is stored in memory as “in-transit” or “in-flight” data of the client device102or the application server112. The message400is shown to include the following components:

A message identifier402: a unique identifier that identifies the message400.

A message text payload404: text, to be generated by a user via a user interface of the client device102and that is included in the message400.

A message image payload406: image data, captured by a camera component of a client device102or retrieved from memory of a client device102, and that is included in the message400.

A message video payload408: video data, captured by a camera component or retrieved from a memory component of the client device102and that is included in the message400.

A message audio payload410: audio data, captured by a microphone or retrieved from the memory component of the client device102, and that is included in the message400.

A message annotations412: annotation data (e.g., filters, stickers or other enhancements) that represents annotations to be applied to message image payload406, message video payload408, or message audio payload410of the message400.

A message duration parameter414: parameter value indicating, in seconds, the amount of time for which content of the message (e.g., the message image payload406, message video payload408, message audio payload410) is to be presented or made accessible to a user via the messaging client application104.

A message geolocation parameter416: geolocation data (e.g., latitudinal and longitudinal coordinates) associated with the content payload of the message. Multiple message geolocation parameter416values may be included in the payload, with each of these parameter values being associated with respect to content items included in the content (e.g., a specific image into within the message image payload406, or a specific video in the message video payload408).

A message story identifier418: identifier value identifying one or more content collections (e.g., “stories”) with which a particular content item in the message image payload406of the message400is associated. For example, multiple images within the message image payload406may each be associated with multiple content collections using identifier values.

A message tag420: each message400may be tagged with multiple tags, each of which is indicative of the subject matter of content included in the message payload. For example, where a particular image included in the message image payload406depicts an animal (e.g., a lion), a tag value may be included within the message tag420that is indicative of the relevant animal. Tag values may be generated manually, based on user input, or may be automatically generated using, for example, image recognition.

A message sender identifier422: an identifier (e.g., a messaging system identifier, email address, or device identifier) indicative of a user of the client device102on which the message400was generated and from which the message400was sent.

A message receiver identifier424: an identifier (e.g., a messaging system identifier, email address or device identifier) indicative of a user of the client device102to which the message400is addressed.

The contents (e.g. values) of the various components of message400may be pointers to locations in tables within which content data values are stored. For example, an image value in the message image payload406may be a pointer to (or address of) a location within an image table308. Similarly, values within the message video payload408may point to data stored within a video table310, values stored within the message annotations412may point to data stored in an annotation table312, values stored within the message story identifier418may point to data stored in a story table306, and values stored within the message sender identifier422and the message receiver identifier424may point to user records stored within an entity table302.

FIG. 5is a schematic diagram illustrating an access-limiting process500, in terms of which access to content (e.g., an ephemeral message502, and associated multimedia payload of data) or a content collection (e.g., an ephemeral message story504), may be time-limited (e.g., made ephemeral).

An ephemeral message502is shown to be associated with a message duration parameter506, the value of which determines an amount of time that the ephemeral message502will be displayed to a receiving user of the ephemeral message502by the messaging client application104. In one embodiment, where the messaging client application104is an application client, an ephemeral message502is viewable by a receiving user for up to a maximum of 10 seconds, depending on the amount of time that the sending user specifies using the message duration parameter506.

The ephemeral message502is shown inFIG. 5to be included within an ephemeral message story504(e.g., a personal story, or an event story). The ephemeral message story504has an associated story duration parameter508, a value of which determines a time-duration for which the ephemeral message story504is presented and accessible to users of the messaging system100. The story duration parameter508, for example, may be the duration of a music concert, where the ephemeral message story504is a collection of content pertaining to that concert. Alternatively, a user (either the owning user or a curator user) may specify the value for the story duration parameter508when performing the setup and creation of the ephemeral message story504.

Additionally, each ephemeral message502within the ephemeral message story504has an associated story participation parameter510, a value of which determines the duration of time for which the ephemeral message502will be accessible within the context of the ephemeral message story504. Accordingly, a particular ephemeral message story504may “expire” and become inaccessible within the context of the ephemeral message story504, prior to the ephemeral message story504itself expiring in terms of the story duration parameter508. The story duration parameter508, story participation parameter510, and message receiver identifier424each provide input to a story timer514, which operationally determines, firstly, whether a particular ephemeral message502of the ephemeral message story504will be displayed to a particular receiving user and, if so, for how long. Note that the ephemeral message story504is also aware of the identity of the particular receiving user as a result of the message receiver identifier424.

Accordingly, the story timer514operationally controls the overall lifespan of an associated ephemeral message story504, as well as an individual ephemeral message502included in the ephemeral message story504. In one embodiment, each and every ephemeral message502within the ephemeral message story504remains viewable and accessible for a time-period specified by the story duration parameter508. In a further embodiment, a certain ephemeral message502may expire, within the context of ephemeral message story504, based on a story participation parameter510. Note that a message duration parameter506may still determine the duration of time for which a particular ephemeral message502is displayed to a receiving user, even within the context of the ephemeral message story504. Accordingly, the message duration parameter506determines the duration of time that a particular ephemeral message502is displayed to a receiving user, regardless of whether the receiving user is viewing that ephemeral message502inside or outside the context of an ephemeral message story504.

The ephemeral timer system202may furthermore operationally remove a particular ephemeral message502from the ephemeral message story504based on a determination that it has exceeded an associated story participation parameter510. For example, when a sending user has established a story participation parameter510of 24 hours from posting, the ephemeral timer system202will remove the relevant ephemeral message502from the ephemeral message story504after the specified 24 hours. The ephemeral timer system202also operates to remove an ephemeral message story504either when the story participation parameter510for each and every ephemeral message502within the ephemeral message story504has expired or when the ephemeral message story504itself has expired in terms of the story duration parameter508.

In certain use cases, a creator of a particular ephemeral message story504may specify an indefinite story duration parameter508. In this case, the expiration of the story participation parameter510for the last remaining ephemeral message502within the ephemeral message story504will determine when the ephemeral message story504itself expires. In this case, a new ephemeral message502, added to the ephemeral message story504, with a new story participation parameter510, effectively extends the life of an ephemeral message story504to equal the value of the story participation parameter510.

Responsive to the ephemeral timer system202determining that an ephemeral message story504has expired (e.g., is no longer accessible), the ephemeral timer system202communicates with the messaging system100(and, for example, specifically the messaging client application104) to cause an indicium (e.g., an icon) associated with the relevant ephemeral message story504to no longer be displayed within a user interface of the messaging client application104. Similarly, when the ephemeral timer system202determines that the message duration parameter506for a particular ephemeral message502has expired, the ephemeral timer system202causes the messaging client application104to no longer display an indicium (e.g., an icon or textual identification) associated with the ephemeral message502.

FIG. 6is a block diagram illustrating functional components of the 3D object generation system210that configures the 3D object generation system210to render 3D objects from 2D objects in a 3D space depicted in a live camera feed. The 3D object generation system210is shown as including a rendering module602, a tracking module604, and a disruption detection module606. The various components and modules of the 3D object generation system210may be configured to communicate with each other (e.g., via a bus, shared memory, or a switch). Although not illustrated inFIG. 6, in some embodiments, the 3D object generation system210may include or may be in communication with a camera configured to produce a camera feed comprising image data that includes a sequence of images (e.g., a video).

Any one or more of the components and modules described may be implemented using hardware alone (e.g., one or more of the processors1504(FIG. 15) of a machine) or a combination of hardware and software. For example, any component and modules described of the 3D object generation system210may physically include an arrangement of one or more of the processors1504(e.g., a subset of or among the one or more processors of the machine) configured to perform the operations described herein for that component. As another example, any component and module of the 3D object generation system210may include software, hardware, or both, that configure an arrangement of one or more processors1504(e.g., among the one or more processors of the machine) to perform the operations described herein for that component. Accordingly, different components and modules of the 3D object generation system210may include and configure different arrangements of such processors1504or a single arrangement of such processors1504at different points in time.

Moreover, any two or more components and modules of the 3D object generation system210may be combined into a single component or module, and the functions described herein for a single component or module may be subdivided among multiple components and modules. Furthermore, according to various example embodiments, components and modules described herein as being implemented within a single machine, database, or device may be distributed across multiple machines, databases, or devices.

The tracking system604may comprise a first tracking sub-system604A, a second tracking sub-system604B, and a third tracking sub-system604C. Each tracking sub-system tracks the position of a 3D object within the 3D space based on a set of tracking indicia.

Tracking systems are subject to frequent tracking failure due to environmental conditions, user actions, unanticipated visual interruption between camera and object/scene being tracked, and so forth. Traditionally, such tracking failures would cause a disruption in the presentation of virtual objects in a 3D space. For example, a virtual object may disappear or otherwise behave erratically, thereby interrupting the illusion of the virtual object being presented within the 3D space. This undermines the perceived quality of the 3D experience as a whole.

Traditional tracking systems rely on a single approach (Natural Feature Tracking (NFT), Simultaneous Localization and Mapping (SLAM), Gyroscopic, etc.) that each have breaking points in real-world usage due to inaccurate sensor data, movement, loss or occlusion of visual marker, or dynamic interruptions to a scene. Further, each approach may have individual limitations in capability. For example, a gyroscopic tracking system can only track items with 3DoF. Further, utilization of a single tracking system provides inaccurate or unstable position estimation, due to inherent limitations of each individual system. For example, an NET system may not provide sufficient pitch, yaw, or roll estimation due to the inaccuracies of visual tracking alone, while gyroscopic tracking systems provide inaccurate translation (up, down, left, right).

To address the foregoing issues with traditional tracking systems, the 3D object generation system210comprises multiple redundant tracking sub-systems604A-C that enable seamless transitions between tracking sub-systems. The multiple redundant tracking sub-systems604A-C address the issues with traditional tracking systems by merging multiple tracking approaches into a single tracking system604. The tracking system604is able to combine 6DoF and 3DoF tracking techniques through combining and transitioning between multiple tracking systems based on the availability of tracking indicia (e.g., roll, pitch, yaw, natural features, etc.) tracked by the tracking systems. Thus, as the indicia tracked by any one tracking system becomes unavailable, the 3D object generation system210seamlessly switches between tracking in 6DoF and 3DoF, thereby providing the user with an uninterrupted experience. For example, in the case of visual tracking systems (e.g., NFT, SLAM), tracking indicia typically analyzed to determine orientation may be replaced with gyroscopic tracking indicia from a gyroscopic tracking system. This would thereby enable transitioning between tracking in 6Dof and 3DoF based on the availability of tracking indicia.

In some example embodiments, to transition between tracking in 6DoF and 3DoF, the 3D object generation system210gathers and stores tracking indicia within a tracking matrix that includes translation indicia (e.g., up, down, left, right) and rotation indicia (e.g., pitch, yaw, roll). The translation indicia gathered by an NFT system may thereby be extracted from the tracking matrix and utilized when future translation indicia gathered by the NFT system become inaccurate or unavailable. In the meantime, the rotation indicia continue to be provided by the gyroscope. In this way, when the mobile device loses tracking indicia, the tracked objects that are presented in the 3D space will not be changed abruptly at the frame when the tracking indicia are lost. Subsequently, when the target tracking object reappears in the screen, and a new translation T1is obtained, the translation part of the view matrix will then be taking advantage of the new translation T1and use T1-T0as the translation of the view matrix.

The rendering module602of the 3D object generation system210is configured to obtain a 2D object and generate and render virtual 3D objects from the 2D objects in a 3D space captured within a live camera feed produced by a camera. For example, the rendering module602may generate a 3D object based on input of a 2D object received from a user (e.g., a 2D image or text) and render the 3D object in the 3D space captured within the live camera feed. In rendering the 3D object, the 3D object generation system210assigns the 3D object to a position in the 3D space based on a real-world reference surface detected in the 3D space. The rendering module602may simultaneously present the 2D object with the 3D object that has been generated in two different positions. The 2D object may be placed at a first user specified location and the 3D object corresponding to the 2D object may be placed at a second user specified location on the screen.

The 3D object generation system210may thereafter track the position of the 3D object relative to a user device in the 3D space by one or more tracking systems in 6DoF. For example, the one or more tracking systems of the 3D object generation system210may collect and analyze a set of tracking indicia (e.g., roll, pitch, yaw, natural features, etc.) in order to track the position of the 3D object relative to the user device in the 3D space with 6DoF. In such embodiments, the 3D object generation system210may transition between tracking systems based on the availability of the tracked indicia to maintain consistent tracking in 6DoF. In an embodiment where both the 2D and 3D objects are simultaneously presented, the 3D object may be tracked in 3D space while the 2D object remains at a static location and is not tracked and does not move in 3D space. In another embodiment, both the 3D object and the 2D object are both tracked in 3D space relative to each other as the live camera angle moves and changes to modify the presentation of the live camera feed in which the 3D object and 2D object are presented. In some embodiments, the 2D object is removed from display after a specified time interval or is not presented at all after the 3D object is generated and displayed in the camera feed. In such cases, the virtual 3D object, generated from the 2D object, is presented in the camera feed and the 2D object is not presented in the camera feed.

The disruption detection module606monitors tracking indicia to detect disruptions. Upon the disruption detection module606detecting an interruption of one or more indicia, such that tracking in 6DoF becomes unreliable or impossible, the 3D object generation system210transitions to tracking the 3D object in the 3D space in 3DoF in order to prevent an interruption of the display. For example, the 3D object generation system210may transition from a first tracking sub-system604A (or first set of tracking sub-systems among the set of tracking sub-systems) that tracks the object in 6DoF to a second tracking sub-system604B among the set of tracking sub-systems (or second set of tracking sub-systems), wherein the second tracking system is capable of tracking the 3D object with 3DoF in the 3D space, based on the tracking indicia available.

In some example embodiments, the set of tracking systems of the 3D object generation system210comprises a gyroscopic tracking system, an NFT system, and a SLAM tracking system. Each tracking system among the set of tracking systems may analyze tracking indicia to track a position of a virtual object within a 3D space. For example, to track a virtual object with 6DoF, the 3D object generation system210may require at least six tracking indicia to be available. As tracking indicia become obstructed or unavailable for various reasons, the 3D object generation system210may transition between the available tracking systems among the set of tracking systems in order to maintain 6DoF or transition to 3DoF if necessary.

It will be readily appreciated that the 3D object generation system210provides consistent rendered virtual objects (e.g., 3D captions or 3D animations, videos or images) in real-world 3D spaces in a wide variety of environments and situations. In many applications it can be desirable to provide firm consistency for the locations of these virtual objects as one or more users, cameras, or other tracking items move around in the environment. This can involve the recognition and use of a specific fixed reference point (e.g., a fixed surface) in the real-world environment. Not using a fixed reference point or item can result in floating or other undesirable inconsistencies in the rendering and presentation of the virtual objects.

To ensure firm consistency in the location of virtual objects, annotation data in the example form of a presentation lens that is specific for the 3D object tracking and rendering described herein may be employed. In particular, a surface aware lens is a presentation lens that identifies and references a real-world surface (e.g., the ground) for the consistent rendering and presentation of virtual objects in 3D space. The surface aware lens can be a specific portion or submodule within the rendering module602. This surface aware lens of the rendering module602can be configured to recognize a reference surface based on visual camera content and may also utilize other device inputs (e.g., gyroscope, accelerometer, compass) to determine what is an appropriate surface within a 3D space depicted in a live camera feed. Once the reference surface has been determined, then a virtual 3D object can be positioned with respect to that reference surface. In an example, the reference surface in the 3D space is a ground surface. In this example, the 3D object generation system210renders the 3D object at a position in the 3D space such that the 3D object appears to be on or slightly above the 3D space.

FIGS. 7-8are flowcharts illustrating example operations of the 3D object generation system in performing methods700and800for generating a message that includes a 3D object created from a 2D object, according to example embodiments. The methods700and800may be embodied in computer-readable instructions for execution by one or more processors such that the operations of the methods700and800may be performed in part or in whole by the functional components of the 3D object generation system210accordingly, the methods700and800are described below by way of example with reference thereto. It shall be appreciated, however, that at least some of the operations of the methods700and800may be deployed on various other hardware configurations and the methods700and800are not intended to be limited to the 3D object generation system210. Any one of the operations described in connection with methods700and800may be performed in a different order than that shown and described or entirely omitted.

At operation702, the annotation system206receives a first input to activate a 3D object lens. The 3D object lens may be selected from a group of lenses.

At operation704, the 3D object generation system210causes display of an editing interface212on the client device102. The editing interface212enables a user to input a selection of a 2D object and modifications or edits to the 2D object that provide a basis for generating a 3D object from the 2D object. To this end, the editing interface212may include a keyboard or other input mechanism to enable the user to input a selection of 2D content (e.g., the one or more text characters, image selection, video selection, animation selection, and so forth). The 2D content input by the user is a displayed as a 2D overlay on top of a camera feed produced by a camera of the client device102.

At operation706, the 3D object generation system210receives a second input comprising one or more 2D objects input by a user of the client device using the editing interface212.

At operation708, the 3D object generation system210generates one or more 3D objects from the 2D content input by the user via editing interface212. As noted above, a 2D representation of the one or more 2D objects input by the user are displayed in the editing interface212as an overlay on top of a camera feed produced by a camera of the client device102. In an embodiment, the 3D object that is generated from the 2D content is presented simultaneously with the 2D representation of the content. In other embodiments, only the 3D object is presented and the 2D object is removed from display. Examples of the editing interface212are discussed below in connection withFIGS. 12-13.

At operation710, the 3D object generation system210detects a third input and, in response to detecting the third input, the 3D object generation system210causes display of a preview interface214, at operation712. The third input may, for example, include a motion-based input such as a change of orientation of the client device102. For example, if the user is pointing to the camera of the client device102at an upward orientation, the 2D representation of the 2D content is presented in the editing interface212. If the user changes the orientation of the camera to be facing downward, the 3D object generation system210may toggle from displaying the editing interface212to displaying the preview interface214. The preview interface214includes a presentation of a 3D object generated based on the 2D content input by the user. The 3D object is a 3D representation of the 2D content input by the user. The 3D object may be rendered at a position in a 3D space captured in the camera feed that is based on a detected reference surface in the 3D space such as a ground or floor surface.

At operation714, the messaging system100generates a message that includes one or more images with the 3D object applied. In generating the message, the messaging system100may record a video of a user specified or predetermined length (e.g., 3 seconds) that includes one or more images from the camera feed with the 3D object applied. The messaging system100may further apply one or more user specified filters to the recorded image(s) in generating the message.

As shown in the method800ofFIG. 8, the method700may, in some embodiments, include operations802,804,806,808, and810. Consistent with these embodiments, the operations802,804,806,808, and810may be performed as part of operation708(e.g., as a sub-routine or sub-operation) where the 3D object generation system210generates a virtual 3D object from an input 2D object and causes display of a preview interface comprising a presentation of the virtual 3D object within a real-world environment captured within a live or recorded camera feed.

At operation802, the rendering component602receives a 2D image comprising a 2D object. For example, the rendering component602receives, from the editing interface, a 2D image that includes text, an animated character, a person, or any other suitable user generated or pre-generated content. In an embodiment, the image is received as a square image of a predetermined size which includes the object or subject of interest within the square. As an example,FIG. 9shows an input image901(e.g., a square image) that includes an object910within the image. In some embodiments, in response to receiving the input image901, the rendering component602generates a texture map shown inFIGS. 10-11(and described below) representing textures of the object910in the input image.

At operation804, the rendering component602generates, based on the 2D image, an alpha image or mask in which the shape and location of the 2D object is represented in a first pixel color and other content in the 2D image is represented in a second pixel color. Specifically, the rendering component602generates an alpha image or mask in which the foreground content (e.g., the 2D object or target object) is represented in a first pixel color and background content in the 2D image is represented in a second pixel color. For example, the rendering component602processes the received square image901on a pixel by pixel basis. For each pixel, the rendering component602determines whether the pixel is transparent (corresponds to a transparent pixel value) or opaque (corresponds to a pixel value that is different from the transparent pixel value). In response to determining that a pixel is transparent, the rendering component602assigns the transparent pixel a white color. In response to determining that a pixel is opaque, the rendering component602assigns the opaque pixel a black color. The rendering component602generates an alpha image that includes the white and black pixels at the corresponding locations of their corresponding transparent or opaque pixels. An example alpha image920is shown inFIG. 9. As shown, the object910is comprised of opaque pixels and is represented by all black pixels in the alpha image920while pixels outside of the object are transparent and are represented by white pixels in the alpha image920.

At operation806, the rendering component602identifies a contour or outline of the 2D object using the alpha image. For example, the rendering component602identifies a black-white boundary in the alpha image920. Particularly, the rendering component602determines where the pixels in the alpha image transition from black to white. The point of transition is marked with another pixel (e.g., a white or grey pixel) and a contour image930is generated to represent all of the points of transition between white and black pixels. The points of transition correspond to the contour of the 2D object910in the input image901. In some embodiments, the points of transition are normalized to be a smooth path around the 2D object910using a normalizing process.

At operation808, the rendering component602generates a 3D mesh based on the contour of the 2D object. For example, the rendering component602normalizes the contour and identifies segments of the normalized contour of the 2D object in the contour image930. For each segment, the rendering component602generates a polygon edge to include in a 3D mesh corresponding to the 2D object910. In an embodiment, the rendering component602rotates the 2D object910by a predetermined number of degrees (e.g., 20 degrees) around its axis and appends the generated polygon edges around the contour of the 2D object910. This results in a 3D mesh corresponding to the 2D object910as shown in 3D mesh940ofFIG. 9.

At operation810, the rendering component602applies a texture of the 2D object to the 3D mesh to output a 3D object representing the 2D object. For example, the rendering component602retrieves a material that maps an input texture to UV positions of the 3D mesh. As shown inFIG. 10, the material includes a front region1010, a back region1020, an outside region1030and an inside region1040. The front region1010identifies a set of UV positions in the 3D mesh corresponding to the front of the 3D object, the hack region1020identifies a set of UV positions in the 3D mesh corresponding to the back of the 3D object, the outside region1030identifies a set of UV positions in the 3D mesh corresponding to the outside of the 3D object, and the inside region1040identifies a set of UV positions in the 3D mesh corresponding to the inside of the 3D object.

Specifically, the rendering component602uses a UV mapping process to map the regions of the material to corresponding portions of the 3D mesh. Namely, UV mapping is the 3D modelling process of projecting a 2D image to a 3D model's surface for texture mapping. The letters “U” and “V” denote the axes of the 2D texture because “X”, “Y,” and “Z” are already used to denote the axes of the 3D object in model space. For example, the UV mapping process involves assigning pixels in the material portions to surface mappings on the polygon, such as by copying a triangular piece of the material portions and pasting it onto a triangle on the 3D object. UV coordinates (e.g., texture coordinates) can be generated for each vertex in the 3D mesh.

The rendering component602applies a first texture from the front region1010to a front portion of the 3D mesh; applies a second texture from the inside region1040to an inside portion of the 3D mesh; applies a third texture from the outside region1030to an outside portion of the 3D mesh; and applies a fourth texture from the back region1020to a back portion of the 3D mesh. For example, the rendering component602applies a first texture1110(FIG. 11) from the front region1010to a front portion of the 3D mesh; applies a second texture1140from the inside region1040to an inside portion of the 3D mesh; applies a third texture1130from the outside region1030to an outside portion of the 3D mesh; and applies a fourth texture1120from the back region1020to a back portion of the 3D mesh. The first, second, third, and fourth textures are applied by overlaying the first, second, third, and fourth textures over portions of the 3D mesh corresponding to the UV positions indicated by the map of the material.

In some embodiments, at least the first and fourth textures1110and1120for the material are generated from the 2D object provided by the user, and the second and third textures1130and1140are previously generated from a preselected or predetermined pattern. In some embodiments, the pattern for the second and third textures1130and1140is selected based on the type of 2D object that is received. For example, if the 2D object is an image, the second and third textures1130and1140correspond to a first pair of opposite patterns (e.g., lines in a first diagonal direction and lines in an opposite diagonal direction). If the 2D object is text, the second and third textures1130and1140correspond to a second pair of patterns (e.g., hashmarks with different densities). In some embodiments, the fourth texture1120or the texture for the back region1020is generated by generating a mirror image of the first texture1110for the front region1010by mirroring the first texture1110for the front region1010vertically.

In some embodiments, the textures of the material vary over time based on a context of the device on which the 3D object is generated. For example, if the 3D object is generated at a first point in time during the morning hours, brighter color textures can be used and if the same 3D object is generated at a second point in time during the evening hours, darker color textures can be used. In some embodiments, only the first texture from the front region1010and the fourth texture from the back region1020are adjusted based on context of the device. For example, the background shown in the first and fourth textures may vary based on context of the device on which the 3D object is presented. Specifically, if the device is determined to be at a high altitude indicating that the user is flying on an airplane, backgrounds shown in the first and fourth textures can change to illustrate a sky. On the other hand, if the device is determined to be at a low altitude but in a geographical position where a body of water is present indicating that the user is on a boat, backgrounds shown in the first and fourth textures can change to illustrate an ocean.

In some embodiments, the various textures that are applied to the 3D object are presented to the user as the user manipulates the 3D object in 3D space. For example, the 3D object can be presented to the user in the real-world environment depicted in the camera feed. The 3D object can initially be presented in a front-facing arrangement such that the back of the 3D object is not visible. As such, the fourth texture from the back region1020is not visible. The user can move the camera in the real-world around the 3D object and/or can turn or rotate the 3D object about its vertical axis. As the user moves the camera to view the back of the object or as the 3D object is rotated, the fourth texture from the back region1020becomes more and more visible. Once the user rotates the object 180 degrees or walks with the camera around the object to view the back of the object, the fourth texture from the back region1020is completely visible and the first texture from the front region1010is no longer visible. As another example, the user can rotate the 3D object about its horizontal axis or position the camera to view the bottom of the object. In such cases, the second texture from the inside region1040becomes more and more visible until the object is rotated 90 degrees about its horizontal axis of when the camera is positioned to full view of the bottom of the object.

In some embodiments, in addition to, or alternative to, generating the 3D object from the 2D object using the extrusion process in which the 3D object is generated based on the inferred contours of the alpha image and applying a 3D mesh and/or texture to the inferred contours as discussed above, the 3D object can be generated directly from the 2D input image901, directly from the inferred contours, directly from the alpha image, or any combination thereof without applying the 3D mesh. By generating the 3D object directly from the input image901, less computational resources are needed and the computational complexity is reduced. This enhances the battery life and increases the speed at which the 3D object is generated. In some embodiments, the rendering component602determines whether processing capabilities of the client device102exceed a threshold or correspond to a specified minimum set of processing capabilities. For example, a determination can be made as to whether the processor of the client device102has a processing speed that exceeds a certain range and has a certain amount of free and available memory space. If the processing capabilities of the client device102exceed the threshold or correspond to the specified minimum set of processing capabilities, the rendering component602can employ the extrusion process in which the 3D object is generated based on the inferred contours of the 2D alpha image and applying a 3D mesh and/or texture to the inferred contours. If the processing capabilities of the client device102fail to exceed or correspond to the threshold or fail to correspond to the specified minimum set of processing capabilities, the rendering component602can generate the 3D object directly from the 2D input image901using a process that requires less computation resources.

In one embodiment, to generate the 3D object directly from the 2D object, the 2D object910depicted in the image901is rotated in 3D space a specified number of degrees (e.g., 20 degrees) about its vertical axis. Arrays of pixels of specified lengths are extended from a first position (e.g., the front of the 2D object) to a second position (e.g., the back of the object). Each array of pixels replicates the color of the pixel or the pixel value from which the array extends. For example, a first pixel from a top portion of the 2D object has a first value. In such cases, an array that is 20 pixels long or 1 centimeter long can be extended from the first pixel towards the back or into the z-axis. The colors of the 20 pixels in the array are also the first value. A second pixel that is adjacent to the first pixel is then selected and another array that is the same or shorter length can be extended from the second pixel towards the back or into the z-axis. The pixels in this array that extends from the second pixel have the same value as the second pixel. In some embodiments, only a portion of the arrays are visible and as the 3D object is rotated in 3D space until other portions of the other arrays become visible.

In one embodiment, to generate the 3D object directly from the 2D object, directly from the inferred contours, directly from the alpha image, or any combination thereof without applying the 3D mesh, the 2D object910is duplicated a specified number of times. Duplicates of the 2D object are stacked behind the 2D object and edges of the duplicates become visible as the virtual 3D object is rotated or as a camera moves around the object. This is referred to as the stacked 2D duplication process for generating a virtual 3D object. Each duplicate of the 2D object is positioned behind another duplicate and its position in 3D space is offset by a specified number of pixels (e.g., 5 or 10 pixels). In this way, a new duplicate is positioned behind the initial 2D object every specified number of pixels.

FIG. 13shows an illustrative 3D object in which duplicates of the 2D object become visible as the 3D object is rotated about its vertical axis. Specifically, the front of the 2D object is shown by a first instance1310of the 2D object. A space1313is inserted behind the first instance1310that is 5 or 10 pixels deep in 3D space and can be represented by pixels having a grey value, Behind the space1313, a second instance1311of the 2D object is presented in 3D space and becomes visible as the 3D object is rotated about its vertical axis. If the 3D object is manipulated to rotate the 3D object 90 degrees about its vertical axis, outside edges (or pixel values along the edges) of all of the duplicates of the 2D images and the offsets having the gray pixel values become fully visible. As one example, edges of 20 different duplicates of the 2D image become visible in this scenario and 20 gray scale offset pixel values separating the duplicates from each other become visible as the 3D object is rotated about its vertical axis. Thus, rather than using a 3D mesh to extend edges of the 2D object in 3D space in a visually and spatially continuous manner, a virtual 3D object is created using a discontinuous duplication of the 2D objected. In an embodiment, as the number of duplicates that are stacked behind the 2D object in 3D space increases, a 3D object created using the 3D mesh applied to the 2D object contours becomes virtually indistinguishable from the 3D object created using the stacked 2D duplication process.

FIGS. 12-13are interface diagrams that illustrate a user interface provided by the messaging system100, according to some embodiments. The user interface includes a lens carousel from which a user may initiate functionality of the 3D object generation system210through selection of a lens or edit icon or option (not shown). Consistent with some embodiments, upon receiving a user selection of the lens or edit icon or option, a user is presented with an editing interface configured for creating and editing a 3D object from a selected 2D object. For example, upon receiving a user selection of the option, the 3D object generation system210may cause display of a user interface illustrated inFIGS. 12-13. As shown inFIG. 12, the user interface presents a 2D object1210selected by the user that provide a basis for a 3D object1220to be rendered within the 3D space.

As shown inFIG. 12, upon receiving input from the user of a 2D object1210, the user interface is updated to present a representation of the 2D object1210. A user of the client device102may access a preview interface (e.g., preview interface214) that includes a preview of the 3D object by providing an input such as changing an orientation of the client device102(e.g., changing the orientation of the camera from pointing upward to pointing downward) or by selecting an interface element (e.g., a button) presented within the user interface of the client device102.FIG. 12illustrates an interface that includes a preview of a 3D object1220generated based on the user supplied 2D object1210. Upon detecting a reference surface (e.g., the ground) in the 3D space captured within the camera feed (e.g., based on a change of orientation of the computing device), the 3D object1220based on the 2D object1210is rendered within the 3D space captured within the camera feed. As shown, the 3D object1220is rendered with respect to a reference surface in the 3D space. That is, the 3D object1220, as rendered, is oriented within the 3D space at a position relative to the reference surface (e.g., the ground). Rendering the 3D object1220in this manner makes it appear to be attached to a real-world surface captured within the camera feed.

A 3D object can be rendered within a 3D space at a first position and the 3D object is rendered such that it appears attached to a reference surface (e.g., the ground). Through appropriate interaction with the 3D object (e.g., a select and drag gesture), the user may move the 3D object such that it is rendered at a second position within the 3D space.

A user may change a scale and rotation of the 3D object through appropriate interaction with the 3D object. For example, the user can perform a pinch and rotate gesture with two fingers on an input touch screen display on which the camera feed is displayed to scale and rotate the 3D object on the reference surface without affecting a layout of the 3D object.

Once the user is satisfied with the placement and look of a 3D object, the user may create a message that includes the 3D object and one or more images from the camera feed. For example, the user may use the client device102to record a video in which the 3D object is rendered such that it appears attached to a surface in the video. While recording the video, the 2D object, that is simultaneously presented with the 3D object, can be omitted such that only the 3D object remains visible in the recorded video.

As part of creating the message, the user may be presented with a menu or other interface element that allows the user to select and apply one or more filters to apply to images of the camera feed along with the 3D object rendered in the 3D space captured within the camera view.

FIG. 14is a block diagram illustrating an example software architecture1406, which may be used in conjunction with various hardware architectures herein described.FIG. 14is a non-limiting example of a software architecture and it will be appreciated that many other architectures may be implemented to facilitate the functionality described herein. The software architecture1406may execute on hardware such as machine1500ofFIG. 15that includes, among other things, processors1504, memory1514, and input/output (I/O) components1518. A representative hardware layer1452is illustrated and can represent, for example, the machine1500ofFIG. 15. The representative hardware layer1452includes a processing unit1454having associated executable instructions1404. Executable instructions1404represent the executable instructions of the software architecture1406, including implementation of the methods, components and so forth described herein. The hardware layer1452also includes memory and/or storage modules memory/storage1456, which also have executable instructions1404. The hardware layer1452may also comprise other hardware1458.

In the example architecture ofFIG. 14, the software architecture1406may be conceptualized as a stack of layers where each layer provides particular functionality. For example, the software architecture1406may include layers such as an operating system1402, libraries1420, applications1416, frameworks/middleware1418, and a presentation layer1414. Operationally, the applications1416and/or other components within the layers may invoke API calls1408through the software stack and receive a response1412as in response to the API calls1408. The layers illustrated are representative in nature and not all software architectures have all layers. For example, some mobile or special purpose operating systems may not provide a frameworks/middleware1418, while others may provide such a layer. Other software architectures may include additional or different layers.

The operating system1402may manage hardware resources and provide common services. The operating system1402may include, for example, a kernel1422, services1424, and drivers1426. The kernel1422may act as an abstraction layer between the hardware and the other software layers. For example, the kernel1422may be responsible for memory management, processor management (e.g., scheduling), component management, networking, security settings, and so on. The services1424may provide other common services for the other software layers. The drivers1426are responsible for controlling or interfacing with the underlying hardware. For instance, the drivers1426include display drivers, camera drivers, Bluetooth® drivers, flash memory drivers, serial communication drivers (e.g., Universal Serial Bus (USB) drivers), Wi-Fi® drivers, audio drivers, power management drivers, and so forth depending on the hardware configuration.

The libraries1420provide a common infrastructure that is used by the applications1416and/or other components and/or layers. The libraries1420provide functionality that allows other software components to perform tasks in an easier fashion than to interface directly with the underlying operating system1402functionality (e.g., kernel1422, services1424and/or drivers1426). The libraries1420may include system libraries1444(e.g., C standard library) that may provide functions such as memory allocation functions, string manipulation functions, mathematical functions, and the like. In addition, the libraries1420may include API libraries1446such as media libraries (e.g., libraries to support presentation and manipulation of various media format such as MPREG4, H.264, MP3, AAC, AMR, JPG, PNG), graphics libraries (e.g., an OpenGL framework that may be used to render 2D and 3D in a graphic content on a display), database libraries (e.g., SQLite that may provide various relational database functions), web libraries (e.g., WebKit that may provide web browsing functionality), and the like. The libraries1420may also include a wide variety of other libraries1448to provide many other APIs to the applications1416and other software components/modules.

The frameworks/middleware1418(also sometimes referred to as middleware) provide a higher-level common infrastructure that may be used by the applications1416and/or other software components/modules. For example, the frameworks/middleware1418may provide various GUI functions, high-level resource management, high-level location services, and so forth. The frameworks/middleware1418may provide a broad spectrum of other APIs that may be utilized by the applications1416and/or other software components/modules, some of which may be specific to a particular operating system1402or platform.

The applications1416include built-in applications1438and/or third-party applications1440. Examples of representative built-in applications1438may include, but are not limited to, a contacts application, a browser application, a book reader application, a location application, a media application, a messaging application, and/or a game application. Third-party applications1440may include an application developed using the ANDROID™ or IOS™ software development kit (SDK) by an entity other than the vendor of the particular platform, and may be mobile software running on a mobile operating system such as IOS™, ANDROID™, WINDOWS® Phone, or other mobile operating systems. The third-party applications1440may invoke the API calls1408provided by the mobile operating system (such as operating system1402) to facilitate functionality described herein.

The applications1416may use built in operating system functions (e.g., kernel1422, services1424, and/or drivers1426), libraries1420, and frameworks/middleware1418to create user interfaces to interact with users of the system. Alternatively, or additionally, in some systems interactions with a user may occur through a presentation layer, such as presentation layer1414. In these systems, the application/component “logic” can be separated from the aspects of the application/component that interact with a user.

The machine1500may include processors1504, memory/storage1506, and I/O components1518, which may be configured to communicate with each other such as via a bus1502. In an example embodiment, the processors1504(e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a radio-frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, a processor1508and a processor1512that may execute the instructions1510. The term “processor” is intended to include multi-core processors1504that may comprise two or more independent processors (sometimes referred to as “cores”) that may execute instructions contemporaneously. AlthoughFIG. 15shows multiple processors, the machine1500may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

The memory/storage1506may include a memory1514, such as a main memory, or other memory storage, and a storage unit1516, both accessible to the processors1504such as via the bus1502. The storage unit1516and memory1514store the instructions1510embodying any one or more of the methodologies or functions described herein. The instructions1510may also reside, completely or partially, within the memory1514, within the storage unit1516, within at least one of the processors1504(e.g., within the processor's cache memory), or any suitable combination thereof, during execution thereof by the machine1500. Accordingly, the memory1514, the storage unit1516, and the memory of processors1504are examples of machine-readable media.

Communication may be implemented using a wide variety of technologies. The I/O components1518may include communication components1540operable to couple the machine1500to a network1532or devices1520via coupling1524and coupling1522, respectively. For example, the communication components1540may include a network interface component or other suitable device to interface with the network1532. In further examples, communication components1540may include wired communication components, wireless communication components, cellular communication components, Near Field Communication (NEC) components, Bluetooth® components (e.g., Bluetooth® Low Energy), Wi-Fi® components, and other communication components to provide communication via other modalities. The devices1520may be another machine or any of a wide variety of peripheral devices (e.g., a peripheral device coupled via a USB).

“CLIENT DEVICE” in this context refers to any machine that interfaces to a communications network to obtain resources from one or more server systems or other client devices. A client device may be, but is not limited to, a mobile phone, desktop computer, laptop, PDA, smart phone, tablet, ultra book, netbook, laptop, multi-processor system, microprocessor-based or programmable consumer electronics, game console, set-top box, or any other communication device that a user may use to access a network.