# Physically-based Lighting of 3D Point Clouds for Quality Assessment

Amar Tious, Toinon Vigier, Vincent Ricordel

► To cite this version:

Amar Tious, Toinon Vigier, Vincent Ricordel. Physically-based Lighting of 3D Point Clouds for Quality Assessment. IMX '23: ACM International Conference on Interactive Media Experiences, Jun 2023, Nantes France, France. pp.423-426, 10.1145/3573381.3597231 . hal-04247652

HAL Id: hal-04247652

https://hal.science/hal-04247652v1

Submitted on 18 Oct 2023

**HAL** is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# Physically-based Lighting of 3D Point Clouds for Quality Assessment

Amar Tious

Nantes Université, École Centrale

Nantes, CNRS, LS2N, UMR 6004,
F-44000 Nantes

France

amar.tious@etu.univ-nantes.fr

Toinon Vigier

Nantes Université, École Centrale

Nantes, CNRS, LS2N, UMR 6004,
F-44000 Nantes

France

toinon.vigier@univ-nantes.fr

Vincent Ricordel

Nantes Université, École Centrale

Nantes, CNRS, LS2N, UMR 6004,
F-44000 Nantes

France

vincent.ricordel@univ-nantes.fr

## ABSTRACT

Point clouds are acknowledged as an essential data structure to represent 3D objects in many use cases, notably immersive experience settings such as Virtual, Augmented or Mixed Reality. This work is the first part of a research project on immersive Quality Assessment of point clouds in different lighting. In this report, I focus mainly on the physically-based rendering of such data in Unity 3D, and the impact of point cloud compression when considering various lighting conditions on the objects. These first observations and results will serve in the implementation of a 6DoF immersive experiment setting for subjective quality assessment.

## CCS CONCEPTS

* Computing methodologies → Rendering; Point-based models; Graphics systems and interfaces.

## KEYWORDS

Point Clouds, 3D graphics, Visual Quality Assessment, Unity Game Engine, Physically-Based Render

### ACM Reference Format:

Amar Tious, Toinon Vigier, and Vincent Ricordel. 2023. Physically-based Lighting of 3D Point Clouds for Quality Assessment. In *ACM International Conference on Interactive Media Experiences (IMX '23)*, June 12–15, 2023, Nantes, France. ACM, New York, NY, USA, 4 pages. https://doi.org/10.1145/3573381.3597231

# 1 INTRODUCTION

## 1.1 Point Clouds and Use cases

Point clouds (PC) are the generalization to 3D of the principle of pixels for 2D images. They are sets of thousands to billions (depending on the use case) of points that each have 3D coordinates and various attribute values (color, normals, transparency,...). There is no spatial connection between the points [14].

The use of point clouds to represent 3D objects has many advantages compared to 3D meshes, which are the traditionally used representations in video games, 3D printing and architecture design. Unlike meshes, they have a faster processing potential as they

don't store and process surface related information. And they allow the representation of non-manifold objects [17].

There exist numerous methods to acquire point cloud data that fit different use cases. For instance, LIDAR sensors are used to scan areas or places and get a point cloud representation of the scene. In this work, we focus on objects or people, static or dynamic (3D video rather than single frame). Technologies used for acquisition of such data are plenoptic and RGB-d cameras, camera matrices, or multi-video camera studios.

These generated data are being used in many fields, like 3D cartography and cultural heritage archives for static point clouds, and immersive videos in Augmented or Virtual Reality for dynamic point clouds, including Virtual Reality streaming and telepresence for online communication.

Point cloud data can be stored in several file formats, such as las and las (LiDAR formats), off, pcd or ply. The last one (PLY: Polygon File Format) stores a list of flat point coordinates with a variety of possible properties including color, normals, transparency and others [14]. It can either be ASCII or binary format, which only affects the in-file syntax.

## 1.2 Our goal and contribution

For the last mentioned application case, the compression and rendering of Point cloud objects in immersive scenes is an essential interest for improvement. Just as for 2D content, its volume can be an obstacle to fast and fluid streaming, and therefore affect the user experience. A poor or unrealistic render of an object in a virtual scene, like objects that don't blend in the lighting of the environment, also result in a less immersive experience. These issues can accentuate the Virtual Reality sickness effects.

To address these issues, compression methods inspired by both 2D and 3D standards are used. These methods are lossy, which means that they result in a distorted version of the source object after decompression. To evaluate these methods' impact on users' visual perception in various use cases, studies of quality assessment for compressed point cloud data can help. Therefore the establishment of standards for compression, subjective evaluation methodologies and objective quality metrics are needed. Many studies already investigate these issues and have run subjective evaluations in immersive environments [9, 13], but most were with static participants, 3 degrees of freedom (DOF) rather than 6, and they didn't take realism into account.

The goal of our research is to assess the impact of different lighting conditions on the perceived quality of distorted decoded point clouds in an immersive scene. This paper first focuses on

<footnote>Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s).

IMX '23, June 12–15, 2023, Nantes, France

© 2023 Copyright held by the owner/author(s).

ACM ISBN 979-8-4007-0028-6/23/06.

https://doi.org/10.1145/3573381.3597231</footnote>
<figure><img src="image_2.png" /><figcaption>Figure 1: Shader graph for Physically-based render in Unity.</figcaption></figure>

point cloud compression (PCC) and PC rendering in Unity3D. We present a few first results and observations about physically-based rendering and lighting of decoded compressed point cloud objects in a 3D scene. From now on we will refer to these decoded objects as distorted or compressed PCs).

## 1.3 Research Questions

The questions that our study investigates are the following:

* Do different types of lighting have an impact on visual quality perception of point clouds?

* What methods for subjective and objective quality assessment work best for re-lit point clouds?

# 2 RELATED WORKS

Our focus has drawn us towards 4 main interests for documentation: PCC, PC rendering, and PC Quality Assessment.

## 2.1 Point cloud compression (PCC)

Two methods for point cloud compression have been selected as standards by MPEG [17]. The first is V-PCC (video-based PCC). Originally for use on dynamic point clouds, this method compresses the data using a standard video encoder by computing 2D projections of the PC (depth maps, texture maps, and others depending on attributes) [6]. G-PCC (geometry-based) uses 3D encoding methods such as octrees or trisoups to compress the geometry data of the PC, and other encoders such as RAHT [3] or PredLift [11] to compress the attributes. More recently, learning-based methods for PCC have been an important part of this topic. A well performing and well known model is GeoCNN [16]. It first voxelizes the PC by partitioning it into blocks and then passes it as input to a trained CNN auto-encoder for compression and decompression. This method does not compress the attributes, only the geometry. These 3 compression/decompression methods are used for distorting PCs in the BASICS dataset [1].

## 2.2 Point Cloud Rendering in Unity

Unity3D doesn't have any built-in importer and rendering solution for PCs. Free and efficient packages are available for this purpose. They differ depending on the file format and the type of data. For instance, BAPC [4] is a renderer for very large point clouds, specifically LiDAR scans (.las, .laz files). It uses multiple threads for continuous real time update of the objects. For different use cases, the more widely used PCx [18] parses .ply files and imports them to Unity as contained in either a mesh, a compute buffer or a texture. With a mesh container, it can then render the point cloud object in real time in a scene using various shaders. For a more realistic and immersive render, solutions using visual effects or custom shaders exist to be able to light the PCs cast shadows. M.Pozzi's Master thesis [15] presents a physically based shader graph to color the object in accordance with the scene's lighting, using the points' normals that should be set as attributes. An example of work making use of custom shader for the lighting of PC objects is "Dissolving Realities" [5].

## 2.3 Subjective Quality Assessment

Plenty of studies regarding the subjective quality assessment (QA) of point clouds or 3D content in immersive settings exist, all with various parameters that can still be explored. Nehmé et. al [13] presents a comparison of many methods for subjective QA of 3D objects. Results suggest that Double Stimulus Impairment Scale (DSIS) methodology is most accurate for this type of content. This result was then used in a QA experiment for colored 3D meshes in VR display (3DOF) [12]. For more DOFs, Gutiérrez et. al [7] proposes a subjective experiment in Mixed Reality with 6 DOFs to observe the influence of real life lighting in MR. For PC quality assessment, Liu et. al [9] and Ak et. al [1] collected subjective evaluation data through experiments on PC visualization on 2D display to build new datasets which respectively are the WPC Database and BASICS. Finally, Evangelos et. al [2] conciliates PC Quality Assessment and
<figure><img src="image_3.png" /><figcaption>Figure 2: Example of a point cloud displayed in Unity using PCx's unlit point cloud shader (right) and a Physically-Based shader graph (left).</figcaption></figure>

6 DOF Virtual reality to validate the efficiency of their PointXR toolbox. This last example has most of the characteristics of the study we aim to run, except for the regard on lighting conditions.

# 3 PRELIMINARY WORK

## 3.1 Stating our choices

In this work, we're using point clouds from BASICS [1]. This dataset contains source point clouds, their distorted versions with the compression methods mentioned earlier, and subjective experiment data. We are only interested in the point clouds. As it is currently used as data for an on-going challenge<fnref n="1" />, we only had access to the training set.

Among the available sources, many PCs have normal attributes that are however not encoded/decoded or mapped to the distorted versions. The PCs are either LiDAR scans, multi-video capture of natural objects or people, or converted meshes.

To render the point clouds, we use Unity Game Engine and the aforementioned PCx package since all our data is in ply format. We modified the data parser of this package in order to implement and use the shader (Fig.1) presented in [15], for displaying a realistic reaction to the light sources of the scene by the PC objects, using its normals as attributes.

## 3.2 Subset of data

As the paper is being submitted, we are still working on the preprocessing of the data and preparation of a subset of BASICS. Since we want the shadows on the PC to be fully dependent on the lighting of the Unity scene, we isolate all the sources that have too much shadows embedded in the color attribute. Then, we are not interested in converted meshes for our use case, so we keep only the PCs of natural content. Because the selected data wasn't large and diverse enough, we included single frames of the commonly used dynamic PCs of 8iLabs [8], in order to have more dynamically acquired PCs representing people, and sources from WPC[9] and LS-PCQA[10].

We have a subset of 31 usable sources (18 people, 13 objects) that are all 800k 1.4M vertices (20 40MB), and all have colors and available distorted versions. For those who don't have normals, we computed them.

## 3.3 Computing normals

As well as some sources, some compressed variants don't have decoded or mapped normals, so they're not compatible with our shader. In Fig.2, we show the impact that a classic 45° directional light has on a point cloud rendered with this shader.

To provide a first example of comparison between a rendered source and its distorted variants in Unity, we computed normals for the processed PCs using Meshlab's normals computation filter<fnref n="2" />. This filter returns satisfying results for dynamically acquired PCs of natural content such as people. Fig.3 is a first example of variants of a same PC rendered with a directional light.

However, our aim is to observe the impact of various lighting on a subject's visual quality perception, and for this we need all our stimuli to have normals as attributes. This Meshlab filter is not an ideal solution as it doesn't perform well on all our PCs, notably some distorted versions, and it can't be used in real time applications for such data size (1 million points). The normals represent a third of a PC file's size. And because we would like to observe the impact of attribute compression on the normals as well, it is all relevant to generate our own distorted stimuli with compressed normals.

The different geometry compression methods we will use are V-PCC, G-PCC and a not yet chosen learning-based method. Additionally, we will use compatible attribute coding method that will get us compressed/decoded PCs with normals and colors as attributes.

Judging by the first examples with computed normals for the processed PCs (Fig.3), we can already suggest that adding lighting to a subjective QA experiment could indeed affect the participant's assessment, notably due to how the density of the PC affects the shadow it casts.

# 4 FUTURE WORKS

For after our study is complete and subjective data have been collected, additional comparisons and experiments could be done besides data analysis. Comparing our data to the ground-truth of the source datasets will allow us to observe the real impact of both the immersive environment and added light conditions on visual quality perception.

As done in [7], we could also move on to objective QA and evaluate performance of state of the art metrics for colored PCs with added light conditions. If results are significant, attempting similar experiments with dynamic point clouds rather than static

<footnote n="1">¹https://sites.google.com/view/icip2023-pcvqa-grand-challenge/home</footnote>

<footnote n="2">²https://www.meshlab.net/</footnote>
<figure><img src="image_4.png" /><figcaption>Figure 3: Example of variants of a PC rendered in Unity. All distorted variants have computed normals. From left to right: source (original normals), source (recomputed normals), distorted with V-PCC, distorted with G-PCC, distorted with GeoCNN</figcaption></figure>

would be a step forward to putting our observations in practice for use cases such as telepresence and immersive 3D video streaming. Finally, our interest in compression of normals could be extended to other PC attributes such as reflectance, or generalized to the relevance of data in efficient compression for satisfying decoding results.

These suggestions for future works will be investigated and explored in context of the Phd thesis on visual improvement of decoded point clouds rendering in extended reality that this study is setting up.

## 5 CONCLUSION

We presented our first progress toward our study of Physically-based render of encoded/decoded PCs for subjective quality assessment experiment. This includes selection of a subset of data relevant to our use case from the BASICS dataset, and first attempts to import and render these data in lit Unity3D scenes. So far, we suggest that various lighting settings (orientation, intensity, ...) could have an impact on visual quality perception.

Immediate continuation of this work will aim to generate new compressed/decoded PCs by using attribute coding methods on the normals, in order to compare PCs with distorted normals to PCs with computed normals. We then plan to run visualization tests of our Unity scenes with Virtual Reality HMDs, in order to design and implement a subjective experiment setup.

## REFERENCES

[1] Ali Ak, Emin Zerman, Maurice Quach, Aladine Chetouani, Aljosa Smolic, Giuseppe Valenzise, and Patrick Le Callet. 2023. BASICS: Broad quality Assessment of Static point clouds In Compression Scenarios. https://doi.org/10.48550/arXiv.2302.04796 arXiv:2302.04796 [cs, eess].

[2] Evangelos Alexiou, Nanyang Yang, and Touradj Ebrahimi. 2020. PointXR: A toolbox for visualization and subjective evaluation of point clouds in virtual reality. In *2020 Twelfth International Conference on Quality of Multimedia Experience (QoMEX)*. IEEE, 1–6.

[3] Ricardo L. de Queiroz and Philip A. Chou. 2016. Compression of 3D Point Clouds Using a Region-Adaptive Hierarchical Transform. *IEEE Transactions on Image Processing* 25, 8 (Aug. 2016), 3947–3956. https://doi.org/10.1109/TIP.2016.2575005 Conference Name: IEEE Transactions on Image Processing.

[4] Simon Maximilian Fraiss and Michael Wimmer. 2017. *Rendering Large Point Clouds in Unity*. Bachelor of Science, Medieninformatik and Visual Computing. Teshcnishe Universitat Wien.

[5] Ruben Frosali. 2020. RubenFro - Dissolving Realities. https://rubenfro.com/dissolving-realities

[6] D. Graziosi, O. Nakagami, S. Kuma, A. Zaghetto, T. Suzuki, and A. Tabatabai. 2020. An overview of ongoing point cloud compression standardization activities: video-based (V-PCC) and geometry-based (G-PCC). *APSIPA Transactions on Signal and Information Processing* 9 (2020), e13. https://doi.org/10.1017/ATSIP.2020.12 Publisher: Cambridge University Press.

[7] Jesús Gutiérrez, Toinon Vigier, and Patrick Le Callet. 2020. Quality Evaluation of 3D Objects in Mixed Reality For Different Lighting Conditions, Vol. 2020. https://doi.org/10.2352/ISSN.2470-1173.2020.11.HVEI-128

[8] Maja Krivokuca, Philip A Chou, and Patrick Savill. 2018. 8i voxelized surface light field (8iVSLF) dataset. *ISO/IEC JTC1/SC29/WG11 MPEG*, input document m42914 (2018).

[9] Qi Liu, Honglei Su, Zhengfang Duanmu, Wentao Liu, and Zhou Wang. 2022. Perceptual Quality Assessment of Colored 3D Point Clouds. *IEEE Transactions on Visualization and Computer Graphics* (2022), 1–1. https://doi.org/10.1109/TVCG.2022.3167151 Conference Name: IEEE Transactions on Visualization and Computer Graphics.

[10] Yipeng Liu, Qi Yang, Yiling Xu, and Le Yang. 2023. Point cloud quality assessment: Dataset construction and learning-based no-reference metric. *ACM Transactions on Multimedia Computing, Communications and Applications* 19, 2s (2023), 1–26.

[11] K Mammou, A Tourapis, J Kim, F Robinet, V Valentin, and Y Su. 2018. Lifting scheme for lossy attribute encoding in TMC1. *Document ISO/IEC JTC1/SC29/WG11 m42640, San Diego, CA, US* (2018).

[12] Yana Nehmé, Florent Dupont, Jean-Philippe Farrugia, Patrick Le Callet, and Guillaume Lavoué. 2021. Visual Quality of 3D Meshes With Diffuse Colors in Virtual Reality: Subjective and Objective Evaluation. *IEEE Transactions on Visualization and Computer Graphics* 27, 3 (March 2021), 2202–2219. https://doi.org/10.1109/TVCG.2020.3036153 Conference Name: IEEE Transactions on Visualization and Computer Graphics.

[13] Yana Nehmé, Jean-Philippe Farrugia, Florent Dupont, Patrick Le Callet, and Guillaume Lavoué. 2021. Comparison of Subjective Methods for Quality Assessment of 3D Graphics in Virtual Reality. *ACM Transactions on Applied Perception* 18, 1 (Dec. 2021), 2:1–2:23. https://doi.org/10.1145/3427931

[14] Stuart Perry. 2020. JPEG Pleno point cloud coding common test conditions v3. 2. 87th JPEG Meeting, WG1N87037 (April 2020).

[15] Matteo Pozzi and Marco Gribaudo. 2022. *Physically based rendering of animated point clouds*. Ph. D. Dissertation. ING - Scuola di Ingegneria Industriale e dell'Informazione.

[16] Maurice Quach, Giuseppe Valenzise, and Frederic Dufaux. 2019. Learning Convolutional Transforms for Lossy Point Cloud Geometry Compression. In *2019 IEEE International Conference on Image Processing (ICIP)*. 4320–4324. https://doi.org/10.1109/ICIP.2019.8803413 ISSN: 2381-8549.

[17] Sebastian Schwarz, Marius Preda, Vittorio Baroncini, Madhukar Budagavi, Pablo Cesar, Philip A. Chou, Robert A. Cohen, Maja Krivokuća, Sébastien Lasserre, Zhu Li, Joan Llach, Khaled Mammou, Rafael Mekuria, Ohji Nakagami, Ernes-tasia Siahaan, Ali Tabatabai, Alexis M. Tourapis, and Vladyslav Zakharchenko. 2019. Emerging MPEG Standards for Point Cloud Compression. *IEEE Journal on Emerging and Selected Topics in Circuits and Systems* 9, 1 (March 2019), 133–148. https://doi.org/10.1109/JETCAS.2018.2885981 Conference Name: IEEE Journal on Emerging and Selected Topics in Circuits and Systems.

[18] Keijiro Takahashi. 2023. Pcx - Point Cloud Importer/Renderer for Unity. https://github.com/keijiro/Pcx original-date: 2017-10-24T16:12:04Z.