Streaming light propagation

A method is provided for streaming light propagation with particular application for feature films and other demanding content creation using scenes of high complexity requiring art directed global illumination. By attaching a data recording shader or equivalent functionality to any tracing based renderer that can provide multi-pass global illumination, the complete set of light bounce propagation records and the set of emissive samples for a particular rendering can be recorded to memory or disk. A user may edit the emissive samples to adjust the lighting environment, including modifying light source color and intensity and even moving and adding new emissive samples. To relight the scene, the edited emissive samples are processed through the propagation records using a streaming multiply-and-add operation amenable to high levels of parallelization, avoiding a costly re-rendering of the scene and providing a final quality result in interactive time.

BACKGROUND

Realistic lighting is an important component of high quality computer rendered graphics. By utilizing a renderer employing a global illumination model, scenes can be provided with convincing reflections and shadows, providing the requisite visual detail demanded by feature length animated films and other content. Conventionally, a ray tracing renderer may be utilized to provide global illumination in a simple manner.

SUMMARY

The present disclosure is directed to streaming light propagation, substantially as shown in and/or described in connection with at least one of the figures, as set forth more completely in the claims.

DETAILED DESCRIPTION

With large processing overhead and highly random data access requirements, ray tracing becomes less suitable for complex scenes with larger amounts of data, as required by feature films and other challenging applications. Moreover, to provide lighting environments that are artistically driven and visually attractive, artists and directors require interactive visualization of lighting changes. A conventional ray tracer requires the entire scene to be re-rendered again to show the result of any lighting changes, a time consuming and resource intensive process that may not be reasonably accommodated within a production budget. While techniques such as renderer state caching and screen-space data structures may assist in accelerating the re-rendering process, such approaches are often only limited to specific portions of the scene and can only provide a lower quality visualization compared to a final quality rendering.

Workstation110may be any computing device such as a rackmount server, desktop computer, or mobile computer. User130may utilize input device135, for example a keyboard and mouse, to direct the operation of rendering application120executing in memory114of processor112. Rendering application120may process scene data150received from network140to generate a rendered output image128for output to display118through GPU116. Network140may be a high speed network suitable for high performance computing (HPC), for example a 10 GigE network or an InfiniBand network. Once completed, output image128may also be copied to non-volatile storage, not shown inFIG. 1.

For simplicity, it is assumed that output image128is only a single frame and that object geometry154already includes the positioning of all objects within the scene for the associated frame. However, in alternative implementations, scene data150may further include motion data for object geometry154, in which case several animation frames may be rendered by rendering application120. Moreover, some implementations may render multiple frames of the same scene concurrently, for example to provide alternative camera angles or to provide stereoscopic rendering. Lighting155may include the properties of all light sources within the scene. Textures156may include all textures necessary for object geometry154. Shaders157may include any shaders necessary to correctly shade object geometry154. Other data may also be stored in scene data150, for example virtual camera parameters and camera paths.

As previously discussed, it is desirable to provide realistic lighting for a computer generated graphics rendering, or output image128. While rasterizing renderers can provide high performance, global illumination can only be approximated. For demanding applications such as feature film rendering, high quality global illumination is required from rendering application120.

Accordingly, rendering application120is any type of renderer that can provide high quality global illumination, such as a ray tracing based renderer. For example, rendering application120may be a streaming global illumination renderer, where all the camera rays122necessary for rendering output image128are generated and kept within memory114. Object geometry154is streamed into memory114as individual work units or nodes, with an exemplary geometry node124as shown, processed against camera rays122using other elements of scene data150as desired, and freed from memory114. Since all required processing is completed after freeing the node from memory, each geometry node124of object geometry154needs to be accessed at most once, and may also be skipped if the geometry node is not visible in the current scene. The above streaming of object geometry154is repeated for as many global illumination passes as required, for example 2-4 passes. Since performing only one pass is equivalent to ray casting, at least two passes are required in one configuration.

Since each geometry node124is an individual work unit and can be processed without dependencies from other geometry nodes, servers145a,145b, and145cmay also be utilized for distributed parallel processing. Servers145a,145b, and145cmay contain components similar to those of workstation110. SIMD (single instruction, multiple data) instructions on processor112and shaders on GPU116may be utilized to further enhance parallelism. Hierarchical traversal across camera rays122and object geometry154may also be utilized to reduce the number of intersection comparisons required.

While high quality global illumination can be provided by using a ray tracing based renderer for rendering application120, interactive visualization of lighting changes is still difficult to provide since scene data150must be re-rendered if lighting155is modified. Since the re-rendering process requires significant time and resources, artists and directors cannot quickly visualize different lighting configurations for optimizing artist-directed lighting in a scene. While some techniques are applicable to accelerate the re-rendering process, such techniques often only affect limited portions of the scene and can only provide a lower quality visualization compared to a final quality rendering.

Accordingly, the recording of light propagation data160is proposed for rendering application120. While rendering application120is tracing output image128for the first time, the light propagation records of camera rays122are recorded as propagation records164within light propagation data160. Additionally, all emission samples and radiance samples are tracked and stored as emission samples162and radiance samples163, respectively. While camera rays are utilized inFIG. 1for simplicity, alternative embodiments may also use camera cones or other shapes for cone tracing or other shape tracing. Intermediate sums for each pass of a multi-bounce global illumination rendering may also be stored within light propagation data160for filtering between bounces.

When emission samples162and therefore lighting155is adjusted, then output image128can be reconstructed by streaming emission samples162through propagation records164, bypassing a re-rendering of scene data150. Relighting of scene data150can therefore be carried out orders of magnitude faster than a straightforward re-rendering. Since the streaming of emission samples162through propagation records164is essentially a streaming multiply-and-add operation amenable to parallel processing rather than a recursive algorithm, rendering application120can relight at interactive rates by utilizing parallelism available to processor112and/or GPU116, allowing artists and directors to immediately visualize lighting changes in full final rendering quality.

For example, assuming a target render size of approximately 2 megapixels for high definition or Full HD (1920 by 1080) video, and assuming a desired sampling of 100 samples per pixel to provide sufficient data for filtering, approximately 200 million propagation records are required per global illumination bounce pass. Assuming each record occupies 20 bytes and assuming four (4) global illumination bounce passes, approximately 16 gigabytes of memory is required from memory114, an amount easily allocated for a modern server or even a high-end consumer class desktop computer. If insufficient memory is available, high speed local storage such as solid state disks and/or RAID arrays may also be utilized.

FIG. 2presents an exemplary diagram of a data structure for light propagation data. Light propagation data260ofFIG. 2includes data261a,261b,261c, and261d, which include emission samples262, radiance samples263, records264a,264b,264c, and264d, and pixels229. With respect toFIG. 2, light propagation data260may correspond to light propagation data160fromFIG. 1. Emission samples262may correspond to emission samples162fromFIG. 1. Pixels229may correspond to pixels of output image128ofFIG. 1. It should be noted that the depiction of light propagation data260inFIG. 2is only a schematic simplification as each data set and record group may potentially contain hundreds of millions of records.

Light propagation data260shows an exemplary recording from four (4) global illumination bounce passes. Accordingly, data261dcorresponds to samples from a fourth pass, data261ccorresponds to samples from a third pass, data261bcorresponds to samples from a second pass, and data261acorresponds to samples from a first pass. Data261dcontains only emission samples262as data261dcorresponds to samples from a final global illumination bounce pass. More specifically, since no further bounces are generated on the final pass, all samples must be emissive by definition since they do not rely on other samples. Data261c,261b, and261amay each include a mix of emission samples262and radiance samples263, as shown. Finally, pixels229, which may correspond to pixels of a final output image128, includes only radiance samples263, as the pixels must be derived from the tracing.

Rendering application120can record light propagation data160including emission samples162as emission samples262and propagation records164as records264a,264b,264c, and264d. The remaining radiance samples263can be derived from this minimal data set. However, to support filtering between bounces, the intermediate sums from radiance samples263may be optionally recorded as well. To implement the recording of light propagation data160in rendering application120, shaders157may include a data recording shader executed for each bounce of camera rays122in rendering application120, thereby recording light propagation data160while generating output image128.

More specifically, each of emission samples262and radiance samples263may correspond to a record containing a color value, such as a red, green, blue (RGB) value. Records264a-264dmay associate source points and destination points in scene data150to emission samples262or radiance samples263. The records may also be segmented according to the associated global illumination (GI) bounce pass. For example, data261dand records264dmay be segmented into a data structure corresponding to GI pass #4, whereas data261aand records264amay be segmented into another data structure corresponding to GI pass #1, as shown. The segmentation may be implemented in the data recording shader, as discussed above.

To improve data coherency for multiple relighting operations, data261a,261b,261c, and261dmay be sorted, for example by source point or destination point. Since a large number of records may need to be sorted, GPU116may be utilized for accelerated sorting. For example, the high performance RadixSorting algorithm can sort over 1G keys per second on a modern CUDA compatible GPU. See, “RadixSorting, High performance GPU radix sorting in CUDA”, available from http://code.google.com/p/back40computing/wiki/RadixSorting.

FIG. 3presents an exemplary flowchart illustrating a method for providing streaming light propagation. Flowchart300begins when processor112of workstation110records propagation records164in a scene represented by scene data150(block310). As previously discussed, this may be carried out by attaching a data recording shader within shaders157, causing rendering application120to update propagation records164as camera rays122are bounced in the scene rendering. Rendering application120may also be directly modified to record propagation records164. As previously discussed, as long as rendering application120is based on a tracing renderer providing high quality global illumination, any suitable renderer may be utilized, including cone tracers and others. After the rendering is finished, the recorded propagation records164may appear similar to records264a-264das shown in light propagation data260ofFIG. 2. As previously discussed, the recording of the propagation records may also be segmented according to GI bounce pass, and the records may be sorted by source or destination index for improved data coherency.

Next, processor112of workstation110determines emission samples162in scene data150(block320). Turning toFIG. 2, this is equivalent to determining emission samples262. For example, the data recording shader may be further configured to also update emission samples162if an intersection sample requires no further bounces, indicating a light emission sample. As previously discussed, radiance samples263may also be recorded by the data recording shader to assist in filtering between bounces. Similar to the propagation records, the data samples may also be segmented according to GI bounce pass, as shown inFIG. 2.

Next, processor112of workstation110edits emission samples162, corresponding to emission samples262inFIG. 2(block330). For example, rendering application120may present a graphical user interface on display118, allowing user130to adjust the lighting via input device135. Thus, user130may modify emission samples262, for example by changing light intensity and/or RGB color values. User130may even move or add new light sources to emission samples262, as long as the new or moved light sources do not require absorbing or scattering that would invalidate the other existing records164. To accommodate any newly updated or added light sources in emission samples262, light propagation records164may be intersected with emissive geometry in scene data150to determine all responsive record updates for propagation records164. Accordingly, user130can flexibly adjust the lighting of the scene to produce art driven lighting effects.

Additionally, user130can flexibly generate effects, mattes, and arbitrary output variables (AOVs) by selecting specific paths for modification. As non-limiting examples, paths intersecting with particular objects or geometry, paths hitting particular light sources, and paths within a specific global illumination pass may be targeted. Since all possible paths are recorded in propagation records164, the selection of specific paths for mattes and AOVs is greatly facilitated. Further, user130can specify radiance filters applied to selected paths that may adjust radiance values for specific regions of scene data150. For example, a color correction or color conversion filter may be provided to modify radiance values for a specific object.

Next, processor112of workstation110generates output image128containing pixels229by propagating the edited emission samples162, corresponding to emission samples262, through propagation records164, or records264a-264d(block340). Example pseudocode is as follows:

Using light propagation data260fromFIG. 2as an example, the above pseudocode would first begin by initializing an array D with emission samples262, as edited by user130. This step would already be completed in steps (320) and (330) above. For simplicity, an array D is assumed that has sufficient memory allocation for all possible samples in the scene; however, an actual implementation may use sparse arrays, linked lists, tree hierarchies, hash tables, and/or other data structures. The remaining values in array D, which include values to be populated with radiance samples263, are initialized to a default value, such as zero.

The outer loop iterates from the last GI pass to the first GI pass. Thus, the GI pass index I begins at GI Pass #4(last GI pass) and decrements (I−) until GI pass #1is processed (I>0), after which the loop finishes. The inner loop iterates through each record R within the present segmentation P[I]. For example, since index I begins at GI Pass #4, the inner loop may first begin by processing each record R in segmentation P[4], or records264dinFIG. 2.

The processing for a particular record R proceeds by retrieving the RGB color value for the source point in R (D[R.sourceIndex][I]), multiplying it by the percentage indicated by the propagation amount in R (R.amount), and adding the result to the RGB color value for the destination point in R (D[R.destinationIndex][I−1]). As shown inFIG. 2, D[R.sourceIndex][I] refers to values in data261d, whereas D[R.destinationIndex][I−1] refers to values in data261c. Note that the use of the += operator preserves any existing value that may be at the destination, since multiple records may accumulate to the same destination.

Since there are no data dependencies, and since writes to the same destination are easily resolved by simple addition, the streaming multiply-and-add operation of the inner loop in the above pseudocode algorithm is highly amenable to parallelism available to processor112, GPU116, and servers145a-145c, allowing for fast calculation of pixels229in interactive time. Thus, user130is enabled to adjust, move, or add to emission samples162and quickly observe the resulting lighting changes to output image128, which may be shown on display118. Advantageously, the relighting of output image128can be provided at full final render quality and automatically accounts for all possible lighting effects supported by rendering application120. Alternatively, to provide even faster results for real-time or near real-time feedback, rendering quality may be reduced using approximations or smaller data sets.