Customizable render pipelines using render graphs

Systems, methods, and computer readable media to data drive a render graph are described. A render graph system defines one or more nodes for a render graph and one or more render targets associated with the nodes. The nodes includes one or more functions to define and resolve target handles for identifying render targets. The render graph system defines one or more connections between the nodes and render targets. The connection between the nodes and render targets form the render graph. The render graph system stores the render graph as a data file and converts, with a render graphics API, the data file into a render graph data object. The render graph system performs a frame setup phase the setups the render graph for a frame based on the render graph data object.

BACKGROUND

This disclosure relates generally to the field of graphics processing. More particularly, but not by way of limitation, this disclosure relates to implementing a customizable render pipeline that automatically manages render targets.

Graphics processor units (GPUs) have become important for processing data parallel graphics tasks in today's computers, mobile devices, and other computing systems. Developers have also been taking advantage of a GPU's parallel capabilities by having the GPU execute non-graphics data tasks in a parallel manner. Vendors and standards organizations have created application programming interfaces (APIs) that make executing data-parallel tasks easier to program because of the high level of developer programming interaction. For example, there are a variety of low level APIs (libraries and frameworks) that reside close to graphics hardware and generally employ outputs from higher level APIs. Specifically, the higher level APIs typically prepare program code for an application and presents the program code to the lower level APIs to process.

Today's graphics processing landscape includes improving real-time graphics rendering. To implement real-time graphics rendering, a modern rendering engine generally needs to be flexible enough to allow for custom programming and a level of configurability to form complicated rendering pipelines. Rendering typically start with an application making a graphics change resulting in a change to a scene. To generate a frame for the scene, a rendering engine may employ several rendering passes prior to committing content to the frame buffer. For example, effects may be sequentially applied to a graphic element, such as lighting, shadows, reflections, specular illumination, etc. In another example, multiple rendering passes may be employed for creating pieces or subsets of a single frame to be composited later to form the entire frame. The use of multiple rendering passes could causes latency that varies dependent upon the speed of the system and the complexity and rate of change of the graphics. For example, in a gaming application, the extent and complexity of graphics can be resource demanding and differs from other graphics application (e.g., three dimensional (3D) modeling). Having an API that is flexible enough to produce rendering pipelines that accommodate a variety of systems and/or graphics application may be beneficial in improving processing time and latency.

SUMMARY

In one embodiment, a method to data drive a render graph. The example method defines one or more nodes for a render graph and one or more render targets associated with the nodes. The nodes includes one or more functions to define and resolve target handles for identifying render targets. The example method defines one or more connections between the nodes and render targets. The connection between the nodes and render targets form the render graph. The example method stores the render graph as a data file and converts, with a render graphics API, the data file into a render graph data object. The example method performs a frame setup phase the setups the render graph for a frame based on the render graph data object.

In another embodiment, a system that comprises memory comprising instructions and at least one processor coupled to memory, where the instructions, when executed, causes the at least one processor to define one or more nodes for a render graph and one or more render targets associated with the nodes. The nodes includes one or more functions to define and resolve target handles for identifying render targets. The processor defines one or more connections between the nodes and render targets. The connection between the nodes and render targets form the render graph. The processor stores the render graph as a data file and converts, with a render graphics API, the data file into a render graph data object. The processor performs a frame setup phase the setups the render graph for a frame based on the render graph data object.

In one embodiment, each of the above described methods, and variation thereof, may be implemented as a series of computer executable instructions. Such instructions may use any one or more convenient programming language. Such instructions may be collected into engines and/or programs and stored in any media that is readable and executable by a computer system or other programmable control device. These and other features will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings and claims.

DETAILED DESCRIPTION

The disclosure includes various example embodiments that create a modular and customizable render pipeline that can be authored in code, driven by data, or both. In particular, a render graph application program interface (API) is able to generate both data driven and code driven render graphs by supporting a developer's ability to create rendering pipelines at varying programming levels. At a top programming level, the render graph API is able to generate a render frame with data-driven render graphs by having the developer interface with a visual graph editor. The visual graph editor is a user interface (UI) for a developer to create and author a visual graph representation of a desired render frame and/or render graphs. The visual graph editor can also provide hooks for a developer to attach render graph assets to portions of a render frame. Based on the visual graph representation, the render graph API generates a render graph asset (e.g., data file) that specifies how a backend render engine generates the render graphs. At the next programming level below the top programming level, the render graph API provides a developer access to portions of a backend render engine by allowing the developer to create with code one or more render graphs that form a render frame. For example, the render graph API exposes and handles inputs and/or outputs of render graphs using a render frame program object. At a next lower programming level, the render graph API provides access to the backend render engine so that a developer is able to write code for managing a collection of nodes of the render graph. Each of the nodes consist of a setup and execute function that declare target usage and dependencies (e.g., setup function) and resolves target handles into textures and performs graphics commands (e.g., execute function).

For the purposes of this disclosure, the term “render graph asset” refers to a digital file that specifies how a backend render engine generates a specific render graph. In one or more embodiments, the “render graph asset” is represented as a JavaScript Object Notation (JSON) data file. A “render graph asset” is similar to other graphics-based assets (e.g., mesh asset or texture asset) in that a developer is able to reuse and/or modify the “render graph asset” for desired purposes. As used herein, the term “render graph” represents a collection of nodes that perform render (e.g., shader) and/or compute operations (e.g., compute kernel) for a rendering pipeline. In one or more embodiments, a “render graph” represents a rendering pass that executes the rendering pipeline. Additionally or alternatively, a “render graph” represents a rendering layer that separates different scene objects into separate images

As used herein, the term “render target” refers to an allocation of memory space for storing data related to executing graphics commands on a graphics processor, such as a GPU. For example, the term “render target” refers to any memory space that a processor and/or a graphics processor access and/or modify when creating and executing a graphics command. For example, the term “render target” includes graphics API resources (e.g., Direct3D® resources), such as buffers and textures. Buffers represent an allocation of unformatted memory that can contain data, such as vertex, shader, and compute state data. Textures represents an allocation of memory for storing formatted image data. In one or more embodiments, the “render target” represent temporary buffers for performing one or more graphics processing tasks. For the purposes of this disclosure, the term “target handle” refers to an abstract reference to a “render target.”

FIG. 1is a diagram of a graphics processing flow100where implementations of the present disclosure may operate.FIG. 1illustrates that an application producing graphics102may issue a graphics requests for a frame in a scene that a high level graphics framework104analyzes and processes. Examples of application producing graphics102include gaming applications, 3D modeling applications, web browser applications, and document viewer applications (e.g., portable document format (pdf) viewer). The high level graphics framework104interacts with a low level graphics framework106to manage changes between frames (e.g., movement of graphics on a screen). The low level graphics framework passes the graphics request to a hardware driver108, after which the hardware (e.g., a graphics processor) may process the data and populate the frame buffer. AlthoughFIG. 1does explicitly illustrates this, there are many software paths to a display device including layers and other frameworks not illustrated inFIG. 1, but the general software architecture options are well known in the art.

In one or more embodiments, the low level graphics framework106may be associated with a library that offers granular control of GPU operations. In particular, some embodiments have a low level graphics framework106that has one or more of the following capabilities or features: direct control of GPU state vectors; facilitation of direct determination/selection of the command buffers being submitted to hardware (encoding and submission); ability to delay commit actions (e.g., the ability to delay commitment or commit command buffers in parallel); offers a standard library; and/or provides granular control of the GPU (e.g., control of the organization, processing, and submission of graphics and computation commands, as well as the management of the associated data and resources for these commands). In some embodiments, the low-level graphics framework may be or may include a standard or published API or library. Examples of low-level graphics framework106include Mantle or Direct3D®.

The ability to control the GPU closely through a published low level graphics framework106provides advantages that may facilitate a more orderly rendering path while allowing application102to use a high level graphics framework104to interface with a system's graphics capabilities. In one or more embodiments, the high level graphics framework104represents a render graph API that generates render graph assets based on data-driven and/or a code-driven operations. For each frame of a scene, the render graph API is able to break up rendering into a collection of interconnected render graphs. Each of the render graphs includes a collection of nodes, where each node consist of a setup and execute function associated with a render or computer operation. In some embodiments, the render graphs are data driven, where the high level graphics framework104(e.g., render graph API) analyzes the data and generates graphics commands expressed in a low level GPU interface language, such as graphics commands facilitated by low level graphics framework106(e.g., Direct3D®). Other embodiments, could include code-driven render graphs and/or nodes within a render graph to generate the graphics commands.

FIG. 2illustrates an embodiment of a processor200that processes multiple frames204generated for an application. Processor200represents a programmable hardware device that is able to process data from one or more data sources, such as memory. InFIG. 2, processor200is a general-purpose processor (e.g., a central processing unit (CPU) or microcontroller) that is not customized to perform specific operations (e.g., processes, calculations, functions, or tasks), and instead is built to perform general compute operations. AlthoughFIG. 2illustrates that processor200is a general-purpose processor, other types of processors could include specialized processor customized to perform specific operations (e.g., processes, calculations, functions, or tasks). Examples of specialized processors include GPUs, floating-point processing units (FPUs), DSPs, FPGAs, application-specific integrated circuits (ASICs), and embedded processors (e.g., universal serial bus (USB) controllers).

Processor200encodes and submits graphics commands that render frames204A-204D to a graphics processor (not shown inFIG. 2). The graphics processor is a specialized processor for performing graphics processing operations. Examples of “graphics processors” include a GPU, DSPs, FPGAs, and/or a CPU emulating a GPU. In one or more embodiments, a graphics processor is also able to perform non-specialized operations that a general-purpose processor is able to perform. Examples of these general compute operations are compute commands associated with compute kernels. A compute kernel refers to a program for a graphics processor (e.g., GPU, DSP, or FPGA). In the context of graphics processing operations, programs for a graphics processor are classified as a compute kernel or a shader. A compute kernel refers to a program for a graphics processor that performs general compute operations (e.g., compute graphics commands), and the term shader refers to a program for a graphics processor that performs graphics operations (e.g., render graphics commands). For purposes of this disclosure the term compute kernel, differs and should not be confused with the term kernel or operating system kernel.

Each frame204A-204D is a representation of a single scene that a graphics processor renders. InFIG. 2, the scene content can change from frame204(e.g., frame204A) to frame204(frame204B). In one or more embodiments, to render the changes for each frame204(e.g., frame204B) a graphics processor partitions the frame204by executing multiple rendering passes. By utilizing multiple rendering passes, graphics processor deconstructs each frame204into multiple component images that can be altered independently before recombining them. For example, rendering passes can separate out different features of a scene for a frame (e.g., shadows, highlights, or reflections) into the separate component images.

InFIG. 2, frame204B is divided based on different camera render layers206A and206B, where each camera render layer206corresponds to a specific camera perspective of the scene. A camera perspective may be based on a variety of parameters, such as position, rotation, scale, field of view, and clipping. Having two camera render layers206A and206B does not cause a graphics processor to draw the entire scene twice, but instead, draws the scene according to different camera perspectives. In other words, each camera render layer206outputs scene objects that are visible to its specific camera perspective. Because of the different camera perspective, camera render layer206A can produce one or more scene objects that are not rendered in camera render layer206B and vice versa. The scene objects not rendered represent scene objects that are not viewable according to the camera perspective for camera render layer206B.

Each camera render layer206A and206B can be further divided into different graphics operations (e.g., render operations).FIG. 2illustrates that both camera render layer206A and206B are broken down into a render opaque mesh operation208A and a render skybox operation210A. The render opaque mesh operation208A includes one or more render commands for drawing meshes. Render skybox operation210also includes one or more render commands for drawing skyboxes. For example, the skybox operation210is able to render skyboxes around a scene to create a complex scenery at a horizon according to a specific camera perspective. Persons of ordinary skill in the art are aware that camera render layers206A and206B can include other graphics operations pertinent to rendering a scene for a specific camera perspective.

FIG. 3illustrates an embodiment of a processor200for processing multiple frames204based on a render graph API. As shown inFIG. 3, a render graph API subdivides frame204B into different API program objects. In comparison toFIG. 2,FIG. 3illustrates that render graph API represents frame204B with a render frame program object302and camera render layers206A and206B with render graph program objects304A and304B, respectively. The different graphics operations within a camera render layers206A and206B are represented as render graph nodes306A-306D. With reference toFIG. 2, the render graph API represents the render opaque mesh operations208A and208B into a render graph nodes306A and306C, respectively. Render sykbox operations210A and210B inFIG. 2are represented as render graph nodes306B and306D, respectively.

FIG. 4depicts a graphical representation of a render frame400that includes multiple render graphs402A-402C built using a render graph API. Each render graph402represents a rendering pipeline that a render pass or a render layer is able to execute. InFIG. 4, the render frame400includes three different render graphs402A-402C sequentially connected together. The shadow render graph402A represents a rendering pipeline for a shadow pass; the camera render graph402B represents a rendering pipeline for a camera render layer; and the post process render graph402C represents a rendering pipeline for a post process pass. To generate render frame400, the render graph API connects the shadow render graph402A sequentially to the camera render graph402B, which then connects to the post process render graph402C. In particular, the shadow render graph402A outputs a shadow buffer410A, which then inputs to the camera render graph402B. The camera render graph402B uses the shadow buffer410A to output an OutColor buffer410B, which then becomes an input to the post process render graph402C. The post process render graph402C uses the OutColor buffer410B to output to color buffer410C.

Within each render graph402A-402C includes one or more nodes404A-404I. Each node404represents a graphics operation that performs one or more graphics commands for a given rendering pipeline. UsingFIG. 4as an example, the shadow render graph402A includes a shadow node404A; the camera render graph402B includes opaque node404B, debug node404C, skybox node404D, transparent node404E, and text node404F; and the post process render graph402C includes a bloom downsample node404G, luminance calculation node404H, and a post process combined node404I. The shadow node404A, opaque node404B, and transparent node404E have a mesh node type indicative of graphics operations that generate and encode draw mesh graphics commands. The debug node404C, skybox node404D, and test node404F have a custom node type that performs developer tailored operations that generate graphics commands. As an example, a developer may have written or provided code for that performs a custom graphics operation that generate graphics command for rendering a sky box. The bloom downsample node404G, luminance calculation node404H, and a post process combined node404I represent full screen type nodes that represent graphics operations for generating graphics commands that render to an entire screen.

FIG. 4also illustrates that processing resources for generating render frame400are shown as shaded boxes. The render graphs402A-402C include rendering targets406A-406G that nodes404A-404I may utilize. InFIG. 4, the rendering targets406A-406G represent intermediate memory buffers for storing graphics data prior to outputting graphics data (e.g., pixel data) to a render graph output buffer, such as shadow buffer410A, OutColor buffer410B, and color buffer410C. ForFIG. 4, the shadow node404A generates data for a shadow map406A; nodes404B-404F within camera render graph402B outputs image information (e.g., pixel data) to an output color buffer406B. Depth stencil406C could represent a depth buffer and/or stencil buffer for tracking depth of pixels on the screen and which fragments should be drawn and not drawn. The other rendering targets406E,406F, and406G correspond to post processing buffers. The shadow buffer410A, OutColor buffer410B, and color buffer410C represent processing resources that another render graph402is able to utilize.

In one or more embodiments, a developer uses a visual graph editor to create render frame400. In other words, rather than writing code to generate the render graphs402A-402C, the developer is able define the different rendering pipeline to render a frame using some type of human readable representation (e.g., a visual graphical representation). The visual graph editor can also provide hooks for attaching render graph assets to a render frame400. After a developer generates the human readable representation for render frame400, the render graph API can convert the human readable representation into one or more data files that specifies how data flows through a backend render engine. For example, the render graph API can create a data file for each render graph402. Specifically, the render graph API creates a data file for the shadow render graph402A, another data file for the camera render graph402B, and third data file for the post process render graph402C. Each data file signifies a render graph asset that can be reused and/or modified to render other frames.

The render graph asset is able to define a collection of nodes, render targets, and the connections between nodes and render targets for a render graph. In a data-driven operation, the render graph API compiles the render graph asset to generate a render graph data object that is feed into a backend render engine. Based off the render graph data object, the backend render engine builds a render graph that manages the collection of nodes. Each of the nodes consist of a setup function and an execute function that can be implemented as callback operations using lambda functions. A setup function declares render target usage and dependencies amongst the render graph nodes and render targets, and a execute function resolves target handles into graphics resources (e.g., a texture) and implement certain graphics commands (e.g., draw call for a collected meshes with a given set of materials). In one or more implementations, the backend render engine is able to manage the render target creation and/or memory aliasing for the render targets.

AlthoughFIG. 4illustrates a specific implementation of render frame400, the disclosure is not limited to the specific implementation illustrated inFIG. 4. Persons of ordinary skill in the art are aware that a variety of other rendering passes and/or rendering layers may be created to generate render frame400. The use and discussion ofFIG. 1is only an example to facilitate ease of description and explanation.

FIG. 5represents a render graph system500that employs a render graph API to data-drive and/or code-drive a render graph.FIG. 5illustrates that the render graph system500is logically divided into multiple programming levels that provides a developer different levels of access into the backend render engine501. At a top programming level530, a developer utilizes a visual graph editor to create a human readable representation (e.g., visual graphic representation) for render frames and/or render graphs. At a next lower programming level532, the render graph API allows a developer to write a render graph and/or render frame in code by exposing the input and/or outputs of nodes and render graphs. At programming level534, a developer is able to write code for how the backend render engine501writes pixels to a screen from start to finish. The backend render engine501manages render target creation, aliasing, and/or memory management operations. The render graph API also allows a developer to access a low level graphics API (e.g., Direct3D®) to generate graphics commands for rendering frames.

The top programming level530allows a developer to generate a variety of render graph assets for a frame. InFIG. 5, a visual graph editor allows a developer to generate render graphs for camera component502and light component504. A developer can create a render graph for the camera component502and/or light component504by utilizing the render graph asset pipeline506. The render graph asset pipeline506can create the render graph by having the developer create a human readable representation of the render graph and/or obtain the previously created and saved render graphs. For example, a developer may utilize the visual graph editor to modify already created render graph assets. The render graph asset pipeline506then generates render graph file508for each render graph. In one or more embodiments, the render graph file508is a JSON file that defines the collection of nodes, render targets, and the connections between the nodes and render targets. The render graph API compiles the render graph file508to generate render graph data object510. By generating the render graph data object510, the visual graph editor is able to provide hooks for attaching different render graph assets to a render graph provider and/or other program objects.

To data-drive render graphs, the render graph node program object512can expose nodes to the visual graph editor through introspection. The render graph file508can define the inputs, outputs and settings for the nodes in a render graph. The visual graph editor also exposes hooks to a developer to apply the render graph program object524to a render graph provider program object516with a render graph provider interface. As shown inFIG. 5, by utilizing the render graph provider interface, the render graph provider program object516may provide a type string and a function to return the render graph data object510. Components, such as camera component502may utilize the render graph provider interface to implement a render graph provider program object516and have an render graph asset handle (e.g., an identifier) to render graph file508. By doing so, at runtime, when the camera is rendered, the camera component502is able to provide its own render graph rather than a default one.

At programming level532, the render graph API allows a developer to access portions of the backend render engine501to write a render graph in code. To write a render graph in code, the developer can use render graph node program object512, render frame program object514, and render graph manager program object518. The render graph node program object512acts as an interface that exposes one or more parameters for a node in a render graph. For example, the render graph node program object512exposes the inputs and/or outputs of the node as members and is also the base class for implementing any type of render graph nodes. For example, a render graph node program object512corresponds to a mesh node that emits draw calls for collected meshes with specific materials.

The render frame program object514handles inputs and output between render graphs. Using a shadow operation example, a developer could have a render graph with many shadow casting lights as well as render graphs of different cameras to view the different shadows. Each of the render graphs could have a same collection of nodes to produce an image output of the scene. The render frame program object415allows a developer to break rendering the scene into a collection of the render graphs and interconnects the render graphs. InFIG. 5, a developer can use the render frame program object514to create the render graph builder program object522and manage the render graph builder program object522across render graphs' setup and compilation phases.

The render frame program object514is also able to use the render graph provider program object516. As shown inFIG. 5, a render graph manager program object518may manage when to provide a render graph to process portions of a frame's rendering. To do this, render graph provider program objects516register with the render graph manager program object518. The render graph manager program object518can map a key to identify a render graph provider program object516. The render graph provider program object516can opt in to provide a render graph for any arbitrary chunk of a render frame. For example, a camera can implement graph provider program object516and produce a render graph for the render frame. Recall that having a render graph provider program object516allows a visual graph editor to provide hooks to optionally data-drive the production of render graph program object524.

At top programming level530, the render graph API allows a developer to access portions of the backend render engine501to write code for how the backend render engine501writes pixels to a screen from start to finish. The render graph node interface program object520represents the basic unit of the render graph system500for implementing programming objects. Each of the render graph node interface program object520consists of a setup and an execute function. The setup function allows a developer to declare render graph target526and dependencies. For example, the setup function defines render graph target526to create, read from, and write to. For read operations, the developer uses a target handle to specify the render graph target526to access from. For a write operation, the developer uses a target handle to specify the render graph target526. The execute function resolves the target handles into render graph target526and performs graphics commands. The graphics commands can be added to a command buffer that is eventually submitted to a graphics processor. As show inFIG. 5, the render graph target526corresponds to a graphics API texture resource528.

The render frame program object514provides a code-driven interface for developers that want to work at top programming level530. Specifically, the render frame program object514allows a developer to build and manage the collection of nodes. The render frame program object514provides a built-in node implementation type referenced as a callback render graph node program object. The callback render graph node program object takes a setup and execution function as lambdas so that developers are able to create nodes without having to wrap the nodes with their own render graph node interface implementing types. The callback render graph node program object provides functionality for developers to write a pixel to screen from start to finish. The backend render engine501manages creating the render graph target526and aliasing, but the developer manages what gets rendered. For example, if a developer needs a shadow map, the developer would add a node to a render graph that produces one and uses one. As shown inFIG. 5, to use a render graph program object524, the developer may utilize the render graph builder program object522, which provides an interface for declaring the render graph targets526.

FIG. 6depicts a flowchart illustrating a graphics processing operation600that data-drive a render graph for a render frame. Operation600utilizes a render graph API to generate both data driven and code driven render graphs. The render graph API is able to support a variety of programming levels for interface with a backend render engine. The use and discussion ofFIG. 5is only an example to facilitate explanation and is not intended to limit the disclosure to this specific example. For example, althoughFIG. 6illustrates that the blocks within operation600are implemented in a sequential order, operation600is not limited to this sequential order.

Operation600may start at block602and define one or more nodes and one or more render targets for a render graph. Operation600can define the render graph using a data-driven approach and/or code-driven approach. The render graph represents a render pipeline that can be executed as a render pass or as a render layer. Operation600may then move to block604and store the render graph as a data file (e.g., JSON data file). As a data file, the render graph becomes an asset that allows a developer to reuse or modify the render graph. Afterwards, operation600can proceed to block606and cover the data file into a render graph data object. At block608, operation600performs a frame setup phase that setups the render graph based on the render graph data object. During the setup phase, operation600may declare target usage and dependencies between nodes and render targets.

After completing the setup phase, operation600moves to block610and optimizes the render graph for processing. In one or more embodiments, operation600analyzes target and/or buffer handle dependency graphs to determine whether a render graph includes unused render targets and/or nodes. Operation600may classify unused render targets and/or nodes based on whether their outputs are eventually connected to a frame buffer or other output from the render graph. If the render targets and/or nodes do not affect the output of the render graph, then operation600classifies the render targets and/or nodes as unused. Additionally or alternatively, operation600may also analyze render target usage declared during setup to manage the lifetime of command encoders used to encode graphics commands in a command buffer. Specifically, operation600evaluates whether nodes read from and/or written to are from the same render target. If so, then operation600combines the nodes together to improve the command encoding operation. After optimizing the render graph, operation600moves to block612and performs a frame execute phase that executes the optimized render graph.

FIG. 7is a simplified block diagram of a computing system700that includes render graph API that may correspond to or may be part of a computer and/or any other computing device, such as a workstation, server, mainframe, super computer, and/or portable computing device. In one or more embodiments computing system700represents different types of electronic systems that enable a person to sense and/or interact with various CGR environments. Examples include head mounted systems, projection-based systems, heads-up displays (HUDs), vehicle windshields having integrated display capability, windows having integrated display capability, displays formed as lenses designed to be placed on a person's eyes (e.g., similar to contact lenses), headphones/earphones, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablets, and desktop/laptop computers. A head mounted system may have one or more speaker(s) and an integrated opaque display. Alternatively, a head mounted system may be configured to accept an external opaque display (e.g., a smartphone). The head mounted system may incorporate one or more imaging sensors to capture images or video of the physical environment, and/or one or more microphones to capture audio of the physical environment. Rather than an opaque display, a head mounted system may have a transparent or translucent display. The transparent or translucent display may have a medium through which light representative of images is directed to a person's eyes. The display may utilize digital light projection, organic light emitting diodes (OLEDs), LEDs, uLEDs, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium may be an optical waveguide, a hologram medium, an optical combiner, an optical reflector, or any combination thereof. In one embodiment, the transparent or translucent display may be configured to become opaque selectively. Projection-based systems may employ retinal projection technology that projects graphical images onto a person's retina. Projection systems also may be configured to project virtual objects into the physical environment, for example, as a hologram or on a physical surface.

FIG. 7illustrates that the computing system700comprises a processor702, which may be also be referenced as a CPU. The processor702may communicate (e.g., via a system bus770) and/or provide instructions to other components within the computing system700, such as the input interface704, output interface706, and/or graphics processor712. In one embodiment, the processor702may comprise one or more multi-core processors and/or memory mediums (e.g., cache memory) that function as buffers and/or storage for data. Additionally, processor702may be part of one or more other processing components, such as application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), and/or digital signal processors (DSPs). AlthoughFIG. 7illustrates that processor702may be a single processor, processor702is not so limited and instead may represent multiple processors. The processor702may be configured to implement any of the operations described herein, which include operation600as described inFIG. 6.

FIG. 7illustrates that memory708may be operatively coupled to processor702. Memory708may be a non-transitory medium configured to store various types of data. For example, memory708may include one or more memory devices that comprise secondary storage, read-only memory (ROM), and/or random-access memory (RAM). The secondary storage is typically comprised of one or more disk drives, optical drives, solid-state drives (SSDs), and/or tape drives and is used for non-volatile storage of data. In certain instances, the secondary storage may be used to store overflow data if the allocated RAM is not large enough to hold all working data. The secondary storage may also be used to store programs that are loaded into the RAM when such programs are selected for execution. The ROM is used to store instructions and perhaps data that are read during program execution. The ROM is a non-volatile memory device that typically has a small memory capacity relative to the larger memory capacity of the secondary storage. The RAM is used to store volatile data and perhaps to store instructions.

As shown inFIG. 7, memory708may be used to house the instructions for carrying out various embodiments described herein. In an embodiment, the memory708may comprise a render engine710that may be accessed and implemented by processor702. Additionally or alternatively, render engine710may be stored and accessed within memory embedded in processor702(e.g., cache memory). The render engine710may be configured to provide computer executable instructions used for generating data driven rendering graphs. In one embodiment, the render engine710may be implemented using the render graph system500as shown inFIG. 5and/or operation600as described inFIG. 6. In one embodiment, memory708may interface with a system bus770(e.g., a computer bus) so as to communicate and/or transmit information stored in memory708to processor702and/or graphics processor712during execution of software programs, such as software applications that comprise program code, and/or computer executable process steps that incorporate functionality described herein.

Persons of ordinary skill in the art are aware that software programs may be developed, encoded, and compiled in a variety computing languages for a variety software platforms and/or operating systems and subsequently loaded and executed by processor702. In one embodiment, the compiling process of the software program, may transform program code written in a programming language to another computer language such that the processor702is able to execute the programming code. For example, the compiling process of the software program may generate an executable program that provides encoded instructions (e.g., machine code instructions) for processor702to accomplish specific, non-generic, particular computing functions, such as data-driving a render graph.

After the compiling process, the render engine710may be loaded as computer executable instructions or process steps to processor702from storage (e.g., memory708, storage medium/media, removable media drive, and/or other storage device) and/or embedded within the processor702. Processor702can execute the stored instructions or process steps in order to perform instructions or process steps (e.g., render engine710) to transform computing system700into a non-generic, particular, specially programmed machine or apparatus. Stored data, e.g., data stored by a storage device, can be accessed by processor702during the execution of computer executable instructions or process steps to instruct one or more components within computing system700.

FIG. 7also illustrates that the processor702may be operatively coupled to an input interface704configured to receive image data, and output interface706configured to output and/or display the frames and a graphics processor712to render frames. The input interface704may be configured to obtain image data and/or other sensor-based information via cables, connectors, wireless connections and/or other communication protocols. In one embodiment, the input interface704may be a network interface that comprises multiple ports configured to receive and/or transmit data via a network. In particular, the network interface may transmit the image data via wired links, wireless link, and/or logical links. Other examples of the input interface704may be universal serial bus (USB) interfaces, CD-ROMs, DVD-ROMs and/or connections to one or more sensors. The output interface706may include to one or more connections for a graphic display (e.g., monitors), a printing device that produces hard-copies of the generated results, and/or a plurality of ports that transmit data via cables, connectors, wireless connections, and/or other communication protocols.

Reference in this disclosure to “one embodiment” or to “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the disclosure, and multiple references to “one embodiment” or “an embodiment” should not be understood as necessarily all referring to the same embodiment. The terms “a,” “an,” and “the” are not intended to refer to a singular entity unless explicitly so defined, but include the general class of which a specific example may be used for illustration. The use of the terms “a” or “an” may therefore mean any number that is at least one, including “one,” “one or more,” “at least one,” and “one or more than one.” The term “or” means any of the alternatives and any combination of the alternatives, including all of the alternatives, unless the alternatives are explicitly indicated as mutually exclusive. The phrase “at least one of” when combined with a list of items, means a single item from the list or any combination of items in the list. The phrase does not require all of the listed items unless explicitly so defined.

For purposes of this disclosure, the term “physical environment” refers to a physical world that people can sense and/or interact with without aid of electronic systems. Physical environments, such as a physical park, include physical articles, such as physical trees, physical buildings, and physical people. People can directly sense and/or interact with the physical environment, such as through sight, touch, hearing, taste, and smell.

In contrast to a VR environment, which is designed to be based entirely on computer-generated sensory inputs, the term “mixed reality (MR) environment” refers to a simulated environment that is designed to incorporate sensory inputs from the physical environment, or a representation thereof, in addition to including computer-generated sensory inputs (e.g., virtual objects). On a virtuality continuum, a mixed reality environment is anywhere between, but not including, a wholly physical environment at one end and virtual reality environment at the other end.

For purposes of this disclosure, “an augmented virtuality (AV) environment” refers to a simulated environment in which a virtual or computer generated environment incorporates one or more sensory inputs from the physical environment. The sensory inputs may be representations of one or more characteristics of the physical environment. For example, an AV park may have virtual trees and virtual buildings, but people with faces photorealistically reproduced from images taken of physical people. As another example, a virtual object may adopt a shape or color of a physical article imaged by one or more imaging sensors. As a further example, a virtual object may adopt shadows consistent with the position of the sun in the physical environment.

At least one embodiment is disclosed and variations, combinations, and/or modifications of the implementation(s) and/or features of the implementation(s) made by a person having ordinary skill in the art are within the scope of the disclosure. Alternative implementations that result from combining, integrating, and/or omitting features of the implementation(s) are also within the scope of the disclosure. Where numerical ranges or limitations are expressly stated, such express ranges or limitations may be understood to include iterative ranges or limitations of like magnitude falling within the expressly stated ranges or limitations (e.g., from about 1 to about 10 includes, 2, 3, 4, etc.; greater than 0.10 includes 0.11, 0.12, 0.13, etc.). The use of the term “about” means ±10% of the subsequent number, unless otherwise stated.