Patent ID: 12223625

DETAILED DESCRIPTION OF THE EMBODIMENTS

In order to make the objects, technical solutions, and advantages of the present invention clearer, the following further describes the present invention in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, and do not limit the protection scope of the present invention.

When the Monte Carlo rendering of the model is performed at a low sampling rate, the Monte Carlo rendering image obtained often has a lot of noise. In order to remove the noise in the Monte Carlo rendering image, the following implementation provides a Generative Adversarial Network (GAN) based on the Monte Carlo rendering image denoising model and the method for establishing the same, a denoising method using the Monte Carlo rendering image denoising model, and a denoising device calling the Monte Carlo rendering image denoising model.

An embodiment provides a method for establishing a Monte Carlo rendering image denoising model based on a Generative Adversarial Network (GAN), as shown inFIG.1andFIG.2, which specifically includes the following processes:

Creating a Training Sample Set

First, using a low sampling rate to perform Monte Carlo rendering on the model to obtain a noisy Monte Carlo rendering image as a noise rendering image. Then, using a high sampling rate to perform Monte Carlo rendering on the same model to obtain a minimal noise Monte Carlo rendering image, and using the Monte Carlo rendering image with very little noise as the target rendering image. Of course, other methods can also be used to denoise the noise rendering image to obtain the target rendering image with the image quality meeting the requirements. The method of obtaining the target rendering image is not limited here.

The goal that the Monte Carlo rendering image denoising model constructed in this embodiment can achieve is to perform a denoising operation on the input noise rendering image, and output the denoising rendering image whose image quality reaches the target rendering image.

In order to improve the denoising ability of the Monte Carlo rendering image denoising model, the present invention also considers adding other auxiliary features as the input of the Monte Carlo rendering image denoising model, so that the Monte Carlo rendering image denoising model can be denoising. Comprehensively combining the noise rendering image features and auxiliary features, and extracting the feature points that can improve the image quality multiple times to form a denoising rendering map. Therefore, when using a low sampling rate to perform Monte Carlo rendering on the model to obtain a noisy Monte Carlo rendering image as a noise rendering image, the auxiliary features corresponding to the noise rendering image are extracted. These auxiliary features include but are not limited to Normal Buffer, Depth Buffer, Material Texture Albedo Buffer.

Therefore, the noise rendering image and the corresponding auxiliary features, and the target rendering image corresponding to the noise rendering image are used as a training sample to construct a training sample set.

Constructing a Generative Adversarial Network (GAN)

The Convolutional Neural Network (CNN) is simply used to denoise the noise rendering image, and the obtained denoising rendering image lacks realism in the details. In order to improve the retention of high-frequency details during denoising, this embodiment constructs Monte Carlo through adversarial learning Rendering image denoising model. Specifically, the constructed Generative Adversarial Network (GAN) includes the Denoising Net and the Critic Net. Among them, the Denoising Net is used to denoise the input noise rendering image and auxiliary features, and the output denoise Noise rendering map, the Critic Net is used to classify the input denoising rendering map and the target rendering map corresponding to the noise rendering map, and output the classification results.

Specifically, the denoising network comprises:An auxiliary graph feature extraction sub-network, the auxiliary graph feature extraction sub-network is a Convolutional Neural Network (CNN) including at least one convolutional layer, and is used to fuse input auxiliary features and output auxiliary feature maps;A rendering map feature extraction sub-network, the rendering map feature extraction sub-network is a Convolutional Neural Network (CNN) including at least one convolutional layer, used for extracting features of a noise rendering map, and outputting a noise feature map;A feature fusion sub-network, the feature fusion sub-network is a neural network that uses residual thought and uses convolutional layers to fuse and extract auxiliary feature maps and noise feature maps.

For the auxiliary graph feature extraction sub-network Encoder Net, it can be a Convolutional Neural Network (CNN) connected in turn with at least two convolutional layers Conv and activation layer RelU. For example, the auxiliary feature fusion network Encoder Net can be as shown inFIG.1(c). The shown Convolutional Neural Network (CNN) specifically includes successively connected Convk3n128s1, Leaky RelU, Conv k1n128s1, Leaky RelU, Conv k1n128s1, Leaky RelU, Conv k1n128s1, Leaky RelU and Conv k1n32s1, where the convolutional kernel of Convk3n128s1 is 3*3. For the convolutional layer with 128 channels and 1 step size, the explanation of other convolutional layers is similar, so it won't be repeated them here.

Specifically, the feature fusion sub-network may comprise:A feature fusion unit, which is used to combine the auxiliary feature map and the noise feature map to output the modulation feature map, specifically including multiple auxiliary feature modulation modules CFM ResBlock, auxiliary feature modulation section CFM and convolutional layer connected in sequence. Among them, the input of the auxiliary feature modulation module CFM Block and the auxiliary feature modulation section CFM are the output of the auxiliary feature map and the previous layer, and the input of the first auxiliary feature modulation module CFM ResBlock is the noise feature map and the auxiliary feature map. The input of convolutional layer is the output of the auxiliary feature modulation section CFM, and the output is the modulation characteristic map;An output unit, which is used to perform feature fusion on the noise feature map output by the feature extraction unit and the modulation feature map output by the modulation unit, that is, the input is the feature map after the noise feature map and the modulation feature map are superimposed, and the output is the denoising rendering image.

Specifically, the auxiliary feature modulation module CFM ResBlock comprise an auxiliary feature modulation section CFM, a convolutional layer, an activation layer, and a superposition operation. Among them, the auxiliary feature modulation section CFM is used to modulate the auxiliary feature and the last output feature, that is, the auxiliary feature modulation section CFM. The input of the feature modulation section CFM includes the auxiliary feature map and the output feature of the previous layer. The superimposition operation is used to superimpose the input of the auxiliary feature modulation module CFM ResBlock and the output of the final convolutional layer.

For example, as shown inFIG.1(b), the auxiliary feature modulation module CFM ResBlock includes the auxiliary feature modulation section CFM, Convk3n64s1, ReLU, the auxiliary feature modulation section CFM, Conv k3n64s1, and the superposition operation ⊕ which are connected in sequence. The input of the auxiliary feature modulation section CFM includes the auxiliary feature map and the output feature of the previous layer. The superimposition operation is used to superimpose the input of the auxiliary feature modulation module CFM ResBlock and the output of Conv k3n64s1.

Among them, the auxiliary feature modulation section CFM includes a convolution layer, a dot multiplication operation, and a superimposition operation. The input of the convolution layer is the auxiliary feature map, and the dot multiplication operation is used to dot multiplicate the output of the convolution layer and the output of the previous layer. The superposition operation is used to superimpose the output of the convolutional layer and the dot multiplication operation, and output the feature map.

For example, as shown inFIG.1(b), the auxiliary feature modulation section CFM includes Conv k1n32s1, Leaky ReLU, Conv k1n64s1, dot multiplication operation ⊙ and superposition operation ⊕, where Conv k1n32s1, Leaky ReLU, and Conv k1n64s1 are connected in sequence. The input of Conv k1n32s1 is the auxiliary feature map, the dot multiplication operation ⊙ refers to the dot multiplication operation of the output of the previous layer and the output γ of Conv k1n64s1. The superposition operation ⊕ refers to the result of the dot multiplication operation and the output β of Conv k1n64s1 Overlay.

Specifically, the fusion unit includes a convolutional layer and an activation layer, and is used to perform feature fusion on the noise feature map output by the feature extraction unit and the modulation feature map output by the modulation unit, and output a denoising feature map. For example, as shown inFIG.1(a), the fusion unit includes Convk3n64s1, ReLU, Convk3n3s1, and ReLU connected in sequence.

Critic Net is a network composed of convolutional layer, BN (Batch Normalization), activation layer and fully connected layer. For example, as shown inFIG.1(d), the discrimination network Critic Net includes successively connected Conv, Leaky ReLU, multiple consecutive extraction units, fully connected layer Dense (100), Leaky ReLU and fully connected layer Dense (1), wherein the extraction unit includes consecutive Conv, BN, and Leaky ReLU. The 100 of the fully connected layer Dense (100) indicates that the output dimension is 100.

Training of Generative Adversarial Network (GAN)

After constructing the Generative Adversarial Network (GAN), the training sample set is used to conduct confrontation training on the Generative Adversarial Network (GAN), and optimize the network parameters of the Generative Adversarial Network (GAN). The role of Denoising Net is to denoise the noise rendering image and generate the denoising rendering image, the purpose of which is to make Critic Net unable to distinguish the denoising rendering image and the target rendering image; while the role of Critic Net is to distinguish as much as possible the visual quality of the denoising rendering image and the target rendering image. Therefore, during the training, the difference between the predicted output of Critic Net and the actual label is used to pass back and update the parameters in the Generative Adversarial Network (GAN) to realize the confrontation training of the generation confrontation network. The whole training is based on the confrontation process so that DenoisingNet and CriticNet can be improved at the same time.

When the parameter tuning is over, extract the Denoising Net determined by the parameters as the Monte Carlo rendering denoising model.

The Monte Carlo rendering image denoising model can achieve the denoising of noisy Monte Carlo rendering images. While achieving a good denoising effect on low-frequency details, it can also significantly improve the retention of high-frequency details to gain more visually-realistic rendering.

Based on the construction of the above-mentioned Monte Carlo rendering image denoising model, the Generative Adversarial Network (GAN) constructed above can also be trained by changing the training samples to obtain a Monte Carlo rendering image denoising model capable of processing other input images.

As is known, Monte Carlo rendering is an improvement of traditional reverse ray tracing, which is still mainly based on the principle of ray tracing. Therefore, when rendering, according to the material difference at the intersection of the first ray and object of path tracing, the rendering process of the rendering engine can be split into the diffuse path rendering process and the specular path rendering process. Separately render using the diffuse path rendering process and the specular path the rendering process to obtain both the Monte Carlo rendering image Pdand the Monte Carlo rendering image Ps.

On this basis, the Monte Carlo rendering image denoising model Mdfor denoising the Monte Carlo rendering image Pdand the Monte Carlo rendering image denoising model Msfor denoising the Monte Carlo rendering image Pscan be obtained.

Specifically, using the Monte Carlo rendering image Pdrendered by the diffuse path rendering process as the noise rendering image Pd(that is, Noisy Diffuse). Using the noise rendering image Pd, the auxiliary feature when generating the noise rendering image Pd, and the target rendering corresponding to the noise rendering map Pdas the training sample to conduct confrontation training on the above-mentioned Generative Adversarial Network (GAN). After confrontation training is completed, extract the denoising network Denoising Net and the auxiliary feature fusion network Encoder Net as the Monte Carlo rendering map denoising model Md.

Using the Monte Carlo rendering image Psrendered by the specular path rendering process as the noise rendering image Ps(that is, Noisy Specular). Using the noise rendering image Ps, the auxiliary feature when generating the noise rendering image Ps, and the target rendering corresponding to the noise rendering map Psas the training sample to conduct confrontation training on the above-mentioned Generative Adversarial Network (GAN). After confrontation training is completed, extract the denoising network Denoising Net and the auxiliary feature fusion network Encoder Net as the Monte Carlo rendering map denoising model Ms.

Another embodiment provides a Monte Carlo rendering image denoising method, as shown inFIG.3, including the following steps:S101: according to the material difference at the intersection point of the first ray and object of the path tracing, the rendering process of the rendering engine is split into a diffuse path rendering process and a specular path rendering process;S102: respectively using the diffuse path rendering process and the specular path rendering process for rendering to obtain the Monte Carlo rendering image Pdand the Monte Carlo rendering image Ps, and generate the auxiliary features of Monte Carlo rendering image Pdand the Monte Carlo rendering image Pscorrespondingly;S103: inputting the Monte Carlo rendering image Pdand the corresponding auxiliary features into the above-mentioned Monte Carlo rendering image denoising model Mdto obtain the denoising rendering image Pd′;S104: inputting the Monte Carlo rendering image Psand the corresponding auxiliary features into the above-mentioned Monte Carlo rendering image denoising model Msto obtain the denoising rendering image Ps′;S105: the denoising rendering image Pd′ and the denoising rendering image Ps′ are merged to obtain a final denoising rendering image.

In the denoising method, the Auxiliary Feature corresponding to the Monte Carlo rendering image Pdand the Monte Carlo rendering image Psincludes, but is not limited to, Normal Buffer, Depth Buffer, and the material texture map Albedo Buffer.

The Monte Carlo rendering image denoising model Mdand the Monte Carlo rendering image denoising model Msare constructed and obtained according to the above construction method, and will not be repeated here.

This denoising method uses the Monte Carlo rendering image denoising models Mdand Ms, which can achieve a rendering effect that can only be achieved by using a lower sampling rate to achieve a higher sampling rate. At the same time, the denoising time is only on the order of one second, which is much less than the rendering time required for multi-sampling (on the order of hundreds to thousands of seconds), which greatly saves rendering time and computing costs, thereby reducing the use of servers and the industry cost of the entire rendering service, and saving resources.

Another embodiment provides a denoising device for Monte Carlo rendering, including a computer memory, a computer processor, and a computer program stored in the computer memory and executable on the computer processor. The above-mentioned Monte Carlo rendering image denoising model Msand the Monte Carlo rendering image denoising model Mdare stored in the computer memory;When the computer processor executes the computer program, the following steps are implemented:According to the material difference at the intersection of the first ray and the object of the path tracing, the rendering process of the rendering engine is split into the diffuse path rendering process and the specular path rendering process;Using the diffuse path rendering process and the specular path rendering process to perform rendering respectively to obtain the Monte Carlo rendering Pdand Monte Carlo rendering Psof low sampling rate, and generate the Auxiliary Features corresponding to Monte Carlo rendering Pdand Monte Carlo rendering Ps;Calling the Monte Carlo rendering image denoising model Mdto denoise the Monte Carlo rendering image Pdand the corresponding auxiliary features to obtain the denoising rendering image Pd′;Calling the Monte Carlo rendering image denoising model Msto denoise the Monte Carlo rendering image Psand the corresponding auxiliary features to obtain the denoising rendering image Ps′;The denoising rendering image Pd′ and the denoising rendering image Ps′ are merged to obtain the final denoising rendering image.

This denoising method uses the Monte Carlo rendering image denoising models Mdand Ms, which can achieve a rendering effect that can only be achieved by using a lower sampling rate to achieve a higher sampling rate. At the same time, the denoising time is only on the order of one second, which is much less than the rendering time required for multi-sampling (on the order of hundreds to thousands of seconds), which greatly saves rendering time and computing costs, thereby reducing the use of servers and the industry cost of the entire rendering service, and saving resources.

The specific implementations described above describe the technical solutions and benefits of the present invention in detail. It should be understood that the above descriptions are only the most preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, complements and equivalent replacements made within the scope shall be included in the protection scope of the present invention.