Query: scene rendering，Vulkan

title: RTX RSim: Accelerated Vulkan Room Response Simulation for Time of Flight Imaging
abstract: Time of Flight camera systems are an essential component in 3D scene analysis and reconstruction for many modern computer vision applications. The development and validation of such systems requires testing in a large variety of scenes and situations. Accurate room impulse response simulation greatly speeds up development and validation, as well as reducing its cost, but large computational overhead has so far limited its applicability. While the overall algorithmic requirements of this simulation differ significantly from 3D rendering, the recently introduced hardware raytracing support in GPUs nonetheless provides an interesting new implementation option. In this paper, we present a new room impulse simulation method, implemented with Vulkan compute shaders and leveraging NVIDIA VKRay hardware raytracing. We also extend this method to asynchronous streaming in order to overcome the limitations of on board GPU memory when simulating very large scenes. Our implementation is, to the best of our knowledge, the first ever application of Vulkan hardware raytracing in a non rendering simulation setting. Compared to a state of the art multicore CPU implementation running on 12 CPU cores, we achieve an overall speedup of factor 20.9 when streaming is not required, and 17.6 with streaming, on a single consumer GPU.

title: GRF: Learning a General Radiance Field for 3D Scene Representation and Rendering
abstract: We present a simple yet powerful implicit neural function that can represent and render arbitrarily complex 3D scenes in a single network only from 2D observations. The function models 3D scenes as a general radiance field, which takes a set of posed 2D images as input, constructs an internal representation for each 3D point of the scene, and renders the corresponding appearance and geometry of any 3D point viewing from an arbitrary angle. The key to our approach is to explicitly integrate the principle of multi view geometry to obtain the internal representations from observed 2D views, guaranteeing the learned implicit representations meaningful and multi view consistent. In addition, we introduce an effective neural module to learn general features for each pixel in 2D images, allowing the constructed internal 3D representations to be remarkably general as well. Extensive experiments demonstrate the superiority of our approach.

title: Object Centric Neural Scene Rendering
abstract: We present a method for composing photorealistic scenes from captured images of objects. Our work builds upon neural radiance fields (NeRFs) which implicitly model the volumetric density and directionally emitted radiance of a scene. While NeRFs synthesize realistic pictures, they only model static scenes and are closely tied to specific imaging conditions. This property makes NeRFs hard to generalize to new scenarios, including new lighting or new arrangements of objects. Instead of learning a scene radiance field as a NeRF does, we propose to learn object centric neural scattering functions (OSFs) a representation that models per object light transport implicitly using a lighting and view dependent neural network. This enables rendering scenes even when objects or lights move, without retraining. Combined with a volumetric path tracing procedure, our framework is capable of rendering both intra and inter object light transport effects including occlusions, specularities, shadows, and indirect illumination. We evaluate our approach on scene composition and show that it generalizes to novel illumination conditions, producing photorealistic, physically accurate renderings of multi object scenes.

title: Scene Aware Audio Rendering via Deep Acoustic Analysis
abstract: We present a new method to capture the acoustic characteristics of real world rooms using commodity devices, and use the captured characteristics to generate similar sounding sources with virtual models. Given the captured audio and an approximate geometric model of a real world room, we present a novel learning based method to estimate its acoustic material properties. Our approach is based on deep neural networks that estimate the reverberation time and equalization of the room from recorded audio. These estimates are used to compute material properties related to room reverberation using a novel material optimization objective. We use the estimated acoustic material characteristics for audio rendering using interactive geometric sound propagation and highlight the performance on many real world scenarios. We also perform a user study to evaluate the perceptual similarity between the recorded sounds and our rendered audio.

title: Neural Inverse Rendering of an Indoor Scene From a Single Image
abstract: Inverse rendering aims to estimate physical attributes of a scene, e.g. reflectance, geometry, and lighting, from image(s) Inverse rendering has been studied primarily for single objects or with methods that solve for only one of the scene attributes. We propose the first learning based approach that jointly estimates albedo, normals, and lighting of an indoor scene from a single image. Our key contribution is the Residual Appearance Renderer (RAR) which can be trained to synthesize complex appearance effects (e.g. inter reflection, cast shadows, near field illumination, and realistic shading) which would be neglected otherwise. This enables us to perform self supervised learning on real data using a reconstruction loss, based on re synthesizing the input image from the estimated components. We finetune with real data after pretraining with synthetic data. To this end, we use physically based rendering to create a large scale synthetic dataset, named SUNCG PBR, which is a significant improvement over prior datasets. Experimental results show that our approach outperforms state of the art methods that estimate one or more scene attributes.

title: Neural scene representation and rendering
abstract: A scene internalizing computer program To train a computer to "recognize" elements of a scene supplied by its visual sensors, computer scientists typically use millions of images painstakingly labeled by humans. Eslami et al. developed an artificial vision system, dubbed the Generative Query Network (GQN) that has no need for such labeled data. Instead, the GQN first uses images taken from different viewpoints and creates an abstract description of the scene, learning its essentials. Next, on the basis of this representation, the network predicts what the scene would look like from a new, arbitrary viewpoint. Science, this issue p. 1204 A computer vision system predicts how a 3D scene looks from any viewpoint after just a few 2D views from other viewpoints. Scene representation the process of converting visual sensory data into concise descriptions is a requirement for intelligent behavior. Recent work has shown that neural networks excel at this task when provided with large, labeled datasets. However, removing the reliance on human labeling remains an important open problem. To this end, we introduce the Generative Query Network (GQN) a framework within which machines learn to represent scenes using only their own sensors. The GQN takes as input images of a scene taken from different viewpoints, constructs an internal representation, and uses this representation to predict the appearance of that scene from previously unobserved viewpoints. The GQN demonstrates representation learning without human labels or domain knowledge, paving the way toward machines that autonomously learn to understand the world around them.

title: Configurable 3D Scene Synthesis and 2D Image Rendering with Per pixel Ground Truth Using Stochastic Grammars
abstract: We propose a systematic learning based approach to the generation of massive quantities of synthetic 3D scenes and arbitrary numbers of photorealistic 2D images thereof, with associated ground truth information, for the purposes of training, benchmarking, and diagnosing learning based computer vision and robotics algorithms. In particular, we devise a learning based pipeline of algorithms capable of automatically generating and rendering a potentially infinite variety of indoor scenes by using a stochastic grammar, represented as an attributed Spatial And Or Graph, in conjunction with state of the art physics based rendering. Our pipeline is capable of synthesizing scene layouts with high diversity, and it is configurable inasmuch as it enables the precise customization and control of important attributes of the generated scenes. It renders photorealistic RGB images of the generated scenes while automatically synthesizing detailed, per pixel ground truth data, including visible surface depth and normal, object identity, and material information (detailed to object parts) as well as environments (e.g. illuminations and camera viewpoints) We demonstrate the value of our synthesized dataset, by improving performance in certain machine learning based scene understanding tasks depth and surface normal prediction, semantic segmentation, reconstruction, etc. and by providing benchmarks for and diagnostics of trained models by modifying object attributes and scene properties in a controllable manner.

title: Vulkan rendering framework for mobile multimedia
abstract: Vulkan is the most recent graphics rendering API, designed mainly for the benefit of gaming and highly intensive workloads where the CPU becomes the bottleneck. While Vulkan is quite amenable to games and other heavy rendering tasks, its use is often discouraged for 2D applications rendering and editing videos with animated effects. In this paper, we present the first Vulkan based animation and effects engine for mobile video rendering to test these claims. We compare our solution with the preloaded video editor applications in mobile phones. We see a significant improvement in all regards, with a 30 FPS playback for 4K videos achieved using 30% less memory and 20% less power.

title: High performance city rendering in Vulkan
abstract: City scale scenes often contain large amounts of geometry and texture that cannot altogether fit on GPU memory. Our ongoing work seek to minimise texture memory usage by streaming only view relevant textures and to improve rendering performance using parallel opportunities offered by Vulkan, the latest generation of graphics API. Our result presents a high performance rendering of a city with streaming textures after CPU and GPU culling.

title: Physically Based Rendering for Indoor Scene Understanding Using Convolutional Neural Networks
abstract: Indoor scene understanding is central to applications such as robot navigation and human companion assistance. Over the last years, data driven deep neural networks have outperformed many traditional approaches thanks to their representation learning capabilities. One of the bottlenecks in training for better representations is the amount of available per pixel ground truth data that is required for core scene understanding tasks such as semantic segmentation, normal prediction, and object boundary detection. To address this problem, a number of works proposed using synthetic data. However, a systematic study of how such synthetic data is generated is missing. In this work, we introduce a large scale synthetic dataset with 500K physically based rendered images from 45K realistic 3D indoor scenes. We study the effects of rendering methods and scene lighting on training for three computer vision tasks: surface normal prediction, semantic segmentation, and object boundary detection. This study provides insights into the best practices for training with synthetic data (more realistic rendering is worth it) and shows that pretraining with our new synthetic dataset can improve results beyond the current state of the art on all three tasks.