Abstract:
A method is provided for streaming light propagation with particular application for feature films and other demanding content creation using scenes of high complexity requiring art directed global illumination. By attaching a data recording shader or equivalent functionality to any tracing based renderer that can provide multi-pass global illumination, the complete set of light bounce propagation records and the set of emissive samples for a particular rendering can be recorded to memory or disk. A user may edit the emissive samples to adjust the lighting environment, including modifying light source color and intensity and even moving and adding new emissive samples. To relight the scene, the edited emissive samples are processed through the propagation records using a streaming multiply-and-add operation amenable to high levels of parallelization, avoiding a costly re-rendering of the scene and providing a final quality result in interactive time.

Description:
BACKGROUND 
     Realistic lighting is an important component of high quality computer rendered graphics. By utilizing a renderer employing a global illumination model, scenes can be provided with convincing reflections and shadows, providing the requisite visual detail demanded by feature length animated films and other content. Conventionally, a ray tracing renderer may be utilized to provide global illumination in a simple manner. 
     SUMMARY 
     The present disclosure is directed to streaming light propagation, substantially as shown in and/or described in connection with at least one of the figures, as set forth more completely in the claims. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  presents an exemplary diagram of a system for providing streaming light propagation; 
         FIG. 2  presents an exemplary diagram of a data structure for light propagation data; and 
         FIG. 3  presents an exemplary flowchart illustrating a method for providing streaming light propagation. 
     
    
    
     DETAILED DESCRIPTION 
     The following description contains specific information pertaining to implementations in the present disclosure. One skilled in the art will recognize that the present disclosure may be implemented in a manner different from that specifically discussed herein. The drawings in the present application and their accompanying detailed description are directed to merely exemplary implementations. Unless noted otherwise, like or corresponding elements among the figures may be indicated by like or corresponding reference numerals. Moreover, the drawings and illustrations in the present application are generally not to scale, and are not intended to correspond to actual relative dimensions. 
     With large processing overhead and highly random data access requirements, ray tracing becomes less suitable for complex scenes with larger amounts of data, as required by feature films and other challenging applications. Moreover, to provide lighting environments that are artistically driven and visually attractive, artists and directors require interactive visualization of lighting changes. A conventional ray tracer requires the entire scene to be re-rendered again to show the result of any lighting changes, a time consuming and resource intensive process that may not be reasonably accommodated within a production budget. While techniques such as renderer state caching and screen-space data structures may assist in accelerating the re-rendering process, such approaches are often only limited to specific portions of the scene and can only provide a lower quality visualization compared to a final quality rendering. 
     Accordingly,  FIG. 1  presents an exemplary diagram of a system for providing streaming light propagation. Diagram  100  of  FIG. 1  includes workstation  110 , display  118 , user  130 , input device  135 , network  140 , servers  145   a ,  145   b  and  145   c , and scene data  150 . Workstation  110  includes processor  112 , memory  114 , and graphics processing unit (GPU)  116 . Memory  114  includes rendering application  120 , camera rays  122 , geometry node  124 , output image  128 , and light propagation data  160 . Light propagation data  160  includes emission samples  162 , radiance samples  163 , and propagation records  164 . Scene data  150  includes object geometry  154 , lighting  155 , textures  156 , and shaders  157 . 
     Workstation  110  may be any computing device such as a rackmount server, desktop computer, or mobile computer. User  130  may utilize input device  135 , for example a keyboard and mouse, to direct the operation of rendering application  120  executing in memory  114  of processor  112 . Rendering application  120  may process scene data  150  received from network  140  to generate a rendered output image  128  for output to display  118  through GPU  116 . Network  140  may be a high speed network suitable for high performance computing (HPC), for example a 10 GigE network or an InfiniBand network. Once completed, output image  128  may also be copied to non-volatile storage, not shown in  FIG. 1 . 
     For simplicity, it is assumed that output image  128  is only a single frame and that object geometry  154  already includes the positioning of all objects within the scene for the associated frame. However, in alternative implementations, scene data  150  may further include motion data for object geometry  154 , in which case several animation frames may be rendered by rendering application  120 . Moreover, some implementations may render multiple frames of the same scene concurrently, for example to provide alternative camera angles or to provide stereoscopic rendering. Lighting  155  may include the properties of all light sources within the scene. Textures  156  may include all textures necessary for object geometry  154 . Shaders  157  may include any shaders necessary to correctly shade object geometry  154 . Other data may also be stored in scene data  150 , for example virtual camera parameters and camera paths. 
     As previously discussed, it is desirable to provide realistic lighting for a computer generated graphics rendering, or output image  128 . While rasterizing renderers can provide high performance, global illumination can only be approximated. For demanding applications such as feature film rendering, high quality global illumination is required from rendering application  120 . 
     Accordingly, rendering application  120  is any type of renderer that can provide high quality global illumination, such as a ray tracing based renderer. For example, rendering application  120  may be a streaming global illumination renderer, where all the camera rays  122  necessary for rendering output image  128  are generated and kept within memory  114 . Object geometry  154  is streamed into memory  114  as individual work units or nodes, with an exemplary geometry node  124  as shown, processed against camera rays  122  using other elements of scene data  150  as desired, and freed from memory  114 . Since all required processing is completed after freeing the node from memory, each geometry node  124  of object geometry  154  needs to be accessed at most once, and may also be skipped if the geometry node is not visible in the current scene. The above streaming of object geometry  154  is repeated for as many global illumination passes as required, for example 2-4 passes. Since performing only one pass is equivalent to ray casting, at least two passes are required in one configuration. 
     Since each geometry node  124  is an individual work unit and can be processed without dependencies from other geometry nodes, servers  145   a ,  145   b , and  145   c  may also be utilized for distributed parallel processing. Servers  145   a ,  145   b , and  145   c  may contain components similar to those of workstation  110 . SIMD (single instruction, multiple data) instructions on processor  112  and shaders on GPU  116  may be utilized to further enhance parallelism. Hierarchical traversal across camera rays  122  and object geometry  154  may also be utilized to reduce the number of intersection comparisons required. 
     While high quality global illumination can be provided by using a ray tracing based renderer for rendering application  120 , interactive visualization of lighting changes is still difficult to provide since scene data  150  must be re-rendered if lighting  155  is modified. Since the re-rendering process requires significant time and resources, artists and directors cannot quickly visualize different lighting configurations for optimizing artist-directed lighting in a scene. While some techniques are applicable to accelerate the re-rendering process, such techniques often only affect limited portions of the scene and can only provide a lower quality visualization compared to a final quality rendering. 
     Accordingly, the recording of light propagation data  160  is proposed for rendering application  120 . While rendering application  120  is tracing output image  128  for the first time, the light propagation records of camera rays  122  are recorded as propagation records  164  within light propagation data  160 . Additionally, all emission samples and radiance samples are tracked and stored as emission samples  162  and radiance samples  163 , respectively. While camera rays are utilized in  FIG. 1  for simplicity, alternative embodiments may also use camera cones or other shapes for cone tracing or other shape tracing. Intermediate sums for each pass of a multi-bounce global illumination rendering may also be stored within light propagation data  160  for filtering between bounces. 
     When emission samples  162  and therefore lighting  155  is adjusted, then output image  128  can be reconstructed by streaming emission samples  162  through propagation records  164 , bypassing a re-rendering of scene data  150 . Relighting of scene data  150  can therefore be carried out orders of magnitude faster than a straightforward re-rendering. Since the streaming of emission samples  162  through propagation records  164  is essentially a streaming multiply-and-add operation amenable to parallel processing rather than a recursive algorithm, rendering application  120  can relight at interactive rates by utilizing parallelism available to processor  112  and/or GPU  116 , allowing artists and directors to immediately visualize lighting changes in full final rendering quality. 
     For example, assuming a target render size of approximately 2 megapixels for high definition or Full HD (1920 by 1080) video, and assuming a desired sampling of 100 samples per pixel to provide sufficient data for filtering, approximately 200 million propagation records are required per global illumination bounce pass. Assuming each record occupies 20 bytes and assuming four (4) global illumination bounce passes, approximately 16 gigabytes of memory is required from memory  114 , an amount easily allocated for a modern server or even a high-end consumer class desktop computer. If insufficient memory is available, high speed local storage such as solid state disks and/or RAID arrays may also be utilized. 
       FIG. 2  presents an exemplary diagram of a data structure for light propagation data. Light propagation data  260  of  FIG. 2  includes data  261   a ,  261   b ,  261   c , and  261   d , which include emission samples  262 , radiance samples  263 , records  264   a ,  264   b ,  264   c , and  264   d , and pixels  229 . With respect to  FIG. 2 , light propagation data  260  may correspond to light propagation data  160  from  FIG. 1 . Emission samples  262  may correspond to emission samples  162  from  FIG. 1 . Pixels  229  may correspond to pixels of output image  128  of  FIG. 1 . It should be noted that the depiction of light propagation data  260  in  FIG. 2  is only a schematic simplification as each data set and record group may potentially contain hundreds of millions of records. 
     Light propagation data  260  shows an exemplary recording from four (4) global illumination bounce passes. Accordingly, data  261   d  corresponds to samples from a fourth pass, data  261   c  corresponds to samples from a third pass, data  261   b  corresponds to samples from a second pass, and data  261   a  corresponds to samples from a first pass. Data  261   d  contains only emission samples  262  as data  261   d  corresponds to samples from a final global illumination bounce pass. More specifically, since no further bounces are generated on the final pass, all samples must be emissive by definition since they do not rely on other samples. Data  261   c ,  261   b , and  261   a  may each include a mix of emission samples  262  and radiance samples  263 , as shown. Finally, pixels  229 , which may correspond to pixels of a final output image  128 , includes only radiance samples  263 , as the pixels must be derived from the tracing. 
     Rendering application  120  can record light propagation data  160  including emission samples  162  as emission samples  262  and propagation records  164  as records  264   a ,  264   b ,  264   c , and  264   d . The remaining radiance samples  263  can be derived from this minimal data set. However, to support filtering between bounces, the intermediate sums from radiance samples  263  may be optionally recorded as well. To implement the recording of light propagation data  160  in rendering application  120 , shaders  157  may include a data recording shader executed for each bounce of camera rays  122  in rendering application  120 , thereby recording light propagation data  160  while generating output image  128 . 
     More specifically, each of emission samples  262  and radiance samples  263  may correspond to a record containing a color value, such as a red, green, blue (RGB) value. Records  264   a - 264   d  may associate source points and destination points in scene data  150  to emission samples  262  or radiance samples  263 . The records may also be segmented according to the associated global illumination (GI) bounce pass. For example, data  261   d  and records  264   d  may be segmented into a data structure corresponding to GI pass # 4 , whereas data  261   a  and records  264   a  may be segmented into another data structure corresponding to GI pass # 1 , as shown. The segmentation may be implemented in the data recording shader, as discussed above. 
     To improve data coherency for multiple relighting operations, data  261   a ,  261   b ,  261   c , and  261   d  may be sorted, for example by source point or destination point. Since a large number of records may need to be sorted, GPU  116  may be utilized for accelerated sorting. For example, the high performance RadixSorting algorithm can sort over 1G keys per second on a modern CUDA compatible GPU. See, “RadixSorting, High performance GPU radix sorting in CUDA”, available from http://code.google.com/p/back40computing/wiki/RadixSorting. 
       FIG. 3  presents an exemplary flowchart illustrating a method for providing streaming light propagation. Flowchart  300  begins when processor  112  of workstation  110  records propagation records  164  in a scene represented by scene data  150  (block  310 ). As previously discussed, this may be carried out by attaching a data recording shader within shaders  157 , causing rendering application  120  to update propagation records  164  as camera rays  122  are bounced in the scene rendering. Rendering application  120  may also be directly modified to record propagation records  164 . As previously discussed, as long as rendering application  120  is based on a tracing renderer providing high quality global illumination, any suitable renderer may be utilized, including cone tracers and others. After the rendering is finished, the recorded propagation records  164  may appear similar to records  264   a - 264   d  as shown in light propagation data  260  of  FIG. 2 . As previously discussed, the recording of the propagation records may also be segmented according to GI bounce pass, and the records may be sorted by source or destination index for improved data coherency. 
     Next, processor  112  of workstation  110  determines emission samples  162  in scene data  150  (block  320 ). Turning to  FIG. 2 , this is equivalent to determining emission samples  262 . For example, the data recording shader may be further configured to also update emission samples  162  if an intersection sample requires no further bounces, indicating a light emission sample. As previously discussed, radiance samples  263  may also be recorded by the data recording shader to assist in filtering between bounces. Similar to the propagation records, the data samples may also be segmented according to GI bounce pass, as shown in  FIG. 2 . 
     Next, processor  112  of workstation  110  edits emission samples  162 , corresponding to emission samples  262  in  FIG. 2  (block  330 ). For example, rendering application  120  may present a graphical user interface on display  118 , allowing user  130  to adjust the lighting via input device  135 . Thus, user  130  may modify emission samples  262 , for example by changing light intensity and/or RGB color values. User  130  may even move or add new light sources to emission samples  262 , as long as the new or moved light sources do not require absorbing or scattering that would invalidate the other existing records  164 . To accommodate any newly updated or added light sources in emission samples  262 , light propagation records  164  may be intersected with emissive geometry in scene data  150  to determine all responsive record updates for propagation records  164 . Accordingly, user  130  can flexibly adjust the lighting of the scene to produce art driven lighting effects. 
     Additionally, user  130  can flexibly generate effects, mattes, and arbitrary output variables (AOVs) by selecting specific paths for modification. As non-limiting examples, paths intersecting with particular objects or geometry, paths hitting particular light sources, and paths within a specific global illumination pass may be targeted. Since all possible paths are recorded in propagation records  164 , the selection of specific paths for mattes and AOVs is greatly facilitated. Further, user  130  can specify radiance filters applied to selected paths that may adjust radiance values for specific regions of scene data  150 . For example, a color correction or color conversion filter may be provided to modify radiance values for a specific object. 
     Next, processor  112  of workstation  110  generates output image  128  containing pixels  229  by propagating the edited emission samples  162 , corresponding to emission samples  262 , through propagation records  164 , or records  264   a - 264   d  (block  340 ). Example pseudocode is as follows: 
     
       
         
               
               
             
           
               
                   
                   
               
             
             
               
                   
                 Initialize array D with edited emission samples; 
               
               
                   
                 For (I = last GI pass; I−−; I &gt; 0) { 
               
               
                   
                  For each record R in segmentation P[I] { 
               
               
                   
                  D[R.destinationIndex][I−1] += D[R.sourceIndex][I] 
               
               
                   
                        * R.amount; 
               
               
                   
                  }} 
               
               
                   
                   
               
             
          
         
       
     
     Using light propagation data  260  from  FIG. 2  as an example, the above pseudocode would first begin by initializing an array D with emission samples  262 , as edited by user  130 . This step would already be completed in steps ( 320 ) and ( 330 ) above. For simplicity, an array D is assumed that has sufficient memory allocation for all possible samples in the scene; however, an actual implementation may use sparse arrays, linked lists, tree hierarchies, hash tables, and/or other data structures. The remaining values in array D, which include values to be populated with radiance samples  263 , are initialized to a default value, such as zero. 
     The outer loop iterates from the last GI pass to the first GI pass. Thus, the GI pass index I begins at GI Pass # 4  (last GI pass) and decrements (I−) until GI pass # 1  is processed (I&gt;0), after which the loop finishes. The inner loop iterates through each record R within the present segmentation P[I]. For example, since index I begins at GI Pass # 4 , the inner loop may first begin by processing each record R in segmentation P[4], or records  264   d  in  FIG. 2 . 
     The processing for a particular record R proceeds by retrieving the RGB color value for the source point in R (D[R.sourceIndex][I]), multiplying it by the percentage indicated by the propagation amount in R (R.amount), and adding the result to the RGB color value for the destination point in R (D[R.destinationIndex][I−1]). As shown in  FIG. 2 , D[R.sourceIndex][I] refers to values in data  261   d , whereas D[R.destinationIndex][I−1] refers to values in data  261   c . Note that the use of the += operator preserves any existing value that may be at the destination, since multiple records may accumulate to the same destination. 
     Since there are no data dependencies, and since writes to the same destination are easily resolved by simple addition, the streaming multiply-and-add operation of the inner loop in the above pseudocode algorithm is highly amenable to parallelism available to processor  112 , GPU  116 , and servers  145   a - 145   c , allowing for fast calculation of pixels  229  in interactive time. Thus, user  130  is enabled to adjust, move, or add to emission samples  162  and quickly observe the resulting lighting changes to output image  128 , which may be shown on display  118 . Advantageously, the relighting of output image  128  can be provided at full final render quality and automatically accounts for all possible lighting effects supported by rendering application  120 . Alternatively, to provide even faster results for real-time or near real-time feedback, rendering quality may be reduced using approximations or smaller data sets. 
     From the above description it is manifest that various techniques can be used for implementing the concepts described in the present application without departing from the scope of those concepts. Moreover, while the concepts have been described with specific reference to certain implementations, a person of ordinary skill in the art would recognize that changes can be made in form and detail without departing from the spirit and the scope of those concepts. As such, the described implementations are to be considered in all respects as illustrative and not restrictive. It should also be understood that the present application is not limited to the particular implementations described herein, but many rearrangements, modifications, and substitutions are possible without departing from the scope of the present disclosure.