Neuro-Symbolic Representations of 3D Scenes using Universal Scene Description Language

Authors: m  shifat hossain
rubel  ahmed
laura  pullum
sumit  jha
rickard  ewetz
Venue: AAAI_org_2024_Workshop_NuCLeaR
Published: 2023-11-29 15:40:16

Abstract: Developing efficient and expressive representations of 3D scenes is a pivotal problem within 3D computer vision. The state-of-the-art approach is based on utilizing 3D point clouds, which is inefficient in data utilization. In this paper, we propose a neuro-symbolic approach leveraging the Universal Scene Description (USD) language. The approach is based on representing 3D scenes using a combination of known objects (symbolic) and 3D point clouds (neural) for the background. We also propose a framework called neuro-symbolic conversion (NSC) for automatically converting 3D scenes into the proposed neuro-symbolic representation. The NSC framework first locates candidate objects in the 3D point cloud representation. Next, the objects are substituted with their compact symbolic representation while considering translations and rotations. The correctness of the substitution is verified by rendering the neuro-symbolic representation and comparing the visual similarity with the original point cloud representation (or RGB-D view). The experimental results demonstrate that our framework is highly accurate in object identification and objection substitution. The neuro-symbolic representations are expected to be useful for downstream tasks such as entity identification, activity recognition, and object tracking.

Main paper: 
 introduction: 
 3D computer vision is vital in machine learning research, significantly contributing to spatial perception and automation, especially in manufacturing, healthcare, and defense. Recent advancements in 3D computer vision include enhanced semantic segmentation (Barbosa and Osório 2023) , robust object pose estimation techniques (Zhu et al. 2022) , and the integration of 3D vision into autonomous systems (Singh and Bankiti 2023) . Traditional explicit 3D representations, like point clouds, meshes, and voxels, encounter challenges in efficiently handling complex and deformable shapes, facing issues in accurately capturing details and managing incomplete or noisy data. Point clouds, utilized for 3D data capture, confront challenges, notably in quantization, leading to precision loss, and their unstructured nature creates compatibility issues with traditional machine learning techniques. Furthermore, innovative methods like voxel-based CNNs (Liu et al. 2019 ) and graph-based approaches (Mirande et al. 2022 ) introduce complexities and high computational overhead when transforming point clouds into structured formats. Symbolic representations of data offer computational efficiency and storage benefits, making them an attractive choice for low-dimensional data. However, their limitations become apparent in high-dimensional scenarios where symbolic approaches may lack expressiveness to effectively capture intricate patterns, presenting challenges in preserving the rich information associated with complex datasets. USD, embraced in production houses like DreamWorks (Blevins and Murray 2018) , becomes an industry standard due to its versatility and user-friendly features.In this paper, we propose a neuro-symbolic approach for representing 3D scenes using USD. The approach integrates known objects (symbolic) with 3D point clouds (neural), resulting in compact and efficient neuro-symbolic representations of intricate 3D scenes, as illustrated in Figure 1 . We also propose a framework called neuro-symbolic conversion (NSC) for converting RGB-D images (or 3D point clouds) into the proposed neuro-symbolic representation using USD. The USD format can render back the original RGB-D image. This bidirectional conversion adeptly represents complex 3D scenes. It demonstrates practical applications by seamlessly substituting objects from a library into the USD format. This framework has the potential for efficient scene manipulation and object recognition. To the best of our knowledge, our neuro-symbolic framework represents a pioneering exploration of USD, offering more efficient and adaptable object representations. Our main contributions can be summarized as follows:• A neuro-symbolic representation of 3D scenes in USD format. The format supports bidirectional conversions between RGB-D images and neuro-symbolic 3D scenes. • Introducing an effective approach called neuro-symbolic conversion (NSC) for identifying and matching objects within a 3D scene using a predefined object library. • The experimental results show that NSC can identify objects with 100% accuracy and substitute each object with more than 90% similarity on average.The organization of this paper is as follows. The remainder of this section reviews some closely related works. Section 2 formulates the problem, and Section 3 describes the proposed 3D scene to USD object conversion and reproduction method. Section 4 presents the experimental results, and the last section concludes the paper.Previous Works. Efficiently representing 3D data is crucial with the growing use of 3D technologies in applications such as virtual reality, mobile mapping, historical artifact scanning, and 3D visualization (Sugimoto et al. 2017; Wandersman 2023 ). The work (Nguyen et al. 2023 ) investigates virtual reality technology for robot environment modeling and presents a method to translate USD-based scene graphs into Knowledge Graphs (KGs). The resulting KG, augmented with dynamic data from a physics simulator, acts as background knowledge for robotic decision-making, demonstrated in a box unpacking scenario. Despite these advances, exploring deep learning with symbolic representations, like USD, remains relatively uncharted. This research avenue holds the promise of seamlessly combining the advantages of compact 3D scene representation in efficient USD data structure, offering potential benefits for machine learning models to operate more efficiently.Learning in a 3D environment is an active research area. The unidirectional transformer-based approach (Hong et al. 2023 ) presents a Large Reconstruction Model, capable of rapidly predicting 3D object models from single input images, trained on an extensive multi-view dataset for enhanced generalizability and performance across various testing scenarios. Neuralangelo (Li et al. 2023 ) combines multiresolution 3D hash grids with neural rendering, utilizing numerical gradients and a coarse-to-fine optimization strategy to achieve superior 3D surface reconstruction from multiview images. Our research focuses on utilizing USD representations of 3D objects to enhance downstream machine learning applications within the 3D environment.2 Neuro-Symbolic Representations of 3D Scenes using USDConversion from a neural representation to a neuro-symbolic representation requires substituting parts of neural representations with symbolic representations. This substitution requires identifying objects of interest from the neural representation and extracting essential information related to the objects in question. The information extracted from the neural images can then be used to reconstruct a symbolic representation of each object and can then be replaced, constructing a neuro-symbolic representation. These neuro-symbolic representations stored in USD formats can be used to store, interpret, analyze, and view neuro-symbolic data. 
problem formulation: 
 Finding the solution to this complex problem can be formulated using the following optimization function:max θ SSIM (I, J θ ) (1)Here, the terms I and J θ are the neural image and neurosymbolic images with different parameters θ, respectively. The neuro-symbolic images are generated based on the detection of an object of interest from the neural image, and then parameters (θ) of the symbolic objects are optimized by maximizing the Structural Similarity Index (SSIM) score. 
overview of proposed methodology: 
 In this section, we provide the details of the neuro-symbolic conversion (NSC) framework. The flow of the framework is shown in Figure 2 The process of seamlessly converting images to and from USD involves harnessing the high-performance capabilities of the USD software platform. In the forward process, RGB and depth images are rendered from existing USD files. This bidirectional conversion process ensures a holistic transformation between neural representations and symbolic descriptions. According to the USD 1.0 specification, a USD can contain different primitive objects. These primitive objects are the nodes that store the mesh and other objects (e.g., lights, cameras, etc.) of a scene. These USD files can then be rendered using a USD rendering engine. We utilize "PointInstancer" to store the neural point cloud data and primitive types to store symbolic library objects in NSC.Conversely, in the reverse process, USD files are generated from RGB-depth images. For instance, RGB and depth Populate object primitives In USD scene Object-wise analysis frames captured from DARPA Airsim are utilized in our specific implementation to form the basis for USD scene generation. This intricate conversion process is performed using USD-core and Kaolin libraries.This step involves semantic segmentation, point-cloud generation, object selection and identification, reconstruction, and USD export. Consider the capture of RGB color images, denoted as I RGB , and depth images, denoted as I Depth , from the Airsim environment. To semantically segment the color images, a proposed open-set semantic segmenter is applied, producing semantic segmentation maps denoted as M Semantic . Concurrently, the depth images are utilized to generate point-cloud information represented as P Cloud .The semantic segmentation maps M Semantic are then intricately projected onto the point-cloud data P Cloud . This projection serves the purpose of symbolically identifying distinct objects within the 3D environment. Mathematically, this projection operation can be expressed as.O Symbolic = Project(M Semantic , P Cloud )(2) Here, O Symbolic is the set of symbolically identified objects from the scene. Subsequently, individual objects are selected from the symbolic identification O Symbolic and undergo direct USD export or reconstruction from the pointcloud information to form 3D meshes. Let M i denote the mesh of the i th object, and O i represent the i th selected object. The reconstruction operation can be represented as:M i = Reconstruct(O i , P Cloud ).The object property extraction from images involves projecting a semantic segmentation map M Semantic to identify object point clouds P Object as shown in Figure 3 . The object-wise point clouds are then analyzed to extract parameters θ = {location, rotation, color, . . .} characterizing each primitive object. These parameters are utilized to place primitives in the scene. Finally, a USD scene is reconstructed by optimizing the parameter set θ using equation 1, combining primitives and their extracted properties. The objective of this step is to ensure the effective extraction and utilization of object properties within the USD environment.The proposed method NSC is outlined in Algorithm 1. The goal is to find the best match between a test image and library objects. Using GradCAM (Selvaraju et al. 2017) , for each object of interest O i in SOL do 3:GradCAM identifies point cloud of O i : P object 4:for each identified point do 5:Remove pixel:I modified ← I test \ P object 6:for each object θ set do 7:Compute SSIM and MSE scores 8:Track scores for each variation 9:end for 10:Select best-matched variation O t 11:Replace removed pixel with O t 12:end for 13:end for 14:return I modified + O t 15: end function the algorithm identifies the pixels in the test image corresponding to the objects of interest. Subsequently, it performs pixel removal and replacement with various library object variations, considering different orientations and colors. The matching process involves computing SSIM and Mean Squared Error (MSE) scores between the modified test image and rendered images with library object variations. The algorithm iterates through possible variations, keeping track of scores for each. The best match for each object is one that has maximum SSIM and minimum MSE scores. The outcome is a neuro-symbolic image showcasing the best-matched variations of the objects of interest. This approach enables efficient estimation of categorical and transformational information for important objects in the scene. 
experimental results: 
 In this section, we validate NSC through two distinct 3D scenarios involving five different scenes featuring a red car and a human as our objects of interest. The objective is to apply NSC to identify point clouds representing these objects, subsequently substituting them with corresponding symbolic library objects. The modified scenes are then compared with the ground truth scenarios on SSIM and MSE metrics to find the best match. Upon finding a highly matching object, transformation information is extracted and stored in USD.Illustrated in Figure 4 , NSC demonstrates notable effectiveness in substituting objects within 3D scenes. The red 1 reports property extractions for the corresponding car object. A high SSIM value, along with very low errors (Err.) in rotation (in radians) and relative depth (in meter) as shown in Scene 1, instills confidence in the identified objects. The scene can be further annotated with the library objects' known attributes, facilitating numerous downstream learning and analysis tasks efficiently. The Accur. (accuracy) column highlights that object identification is correct for all instances (108 out of 108) in this example scenario. A similar trend is observed in MSE and SSIM scores for the human, albeit not as ideal as the car object. Humans are correctly identified in all scenes (72 out of 72) and replaced accurately. Our investigation reveals that the human object's location in the scene relative to the surroundings contributes to poorer MSE and SSIM compared to the car object.NSC iteratively estimates both categorical and transformational information for essential objects, facilitating the update of object attributes within the USD library. The preliminary results affirm the feasibility of NSC in intricate neuro-symbolic AI tasks. 
conclusion and future work: 
 Our work demonstrates a substantial advancement in the neuro-symbolic representation of 3D scenes. The bidirectional image-to-USD conversion, object-wise property extraction, and dynamic library matching highlight the efficiency and adaptability of our approach. In future, we plan to expand our method to enhance the symbolic reasoning of machine learning models, with a specific focus on deeper exploration of inter-symbolic relationships. 
 
 Appendix:  


Decision: I have changed the order of the authors.

Reviews: review: title: title: A Review of NuCLear Submission 21
main_review: main_review: The authors have convinced me that USD is worth investigating for data storage, when AI/ML operations over 3DPC images is being done.  Their "NSC" is apparently a useful way to convert between 3DPC and NS representations.  No concern is given to processing time, suggesting that the ground-truth-preserving and cost-optimized results of NSC are of high enough value in themselves to ignore such overhead initially.  The same is true for the particulars of the NSC method--they are not inevitable, and have cost, but the fact that the NSC includes iterations through an object library, allowing for transformations, seems to make NSC's versatility more prominent than its speed.


What I don't know yet:
* what, if any, selection procedure was used to yield the particular semantic segmenter they used, and/or its invocation parameters?
* what data storage savings are realized by using NS representations (in USD, in this case)?  I can easily infer that they must exist, but since the authors explicitly state that 3DPC representations are inefficient, I'd like to see a contrast.
* how does the computational overhead of using NSC's central algorithm compare with the computational overheads associated with voxel-based CNNs and graph-based approaches?
* what is it about USD that makes it, as opposed to other storage formats, especially suitable for this purpose?  Again, I can see that it probably is (due to extensibility and ecosystem), but all we're told is that USD is considered versatile and user-friendly in "production houses", and has "high-performance capabilities".  I infer that "permits use of things like pointinstancer" is one such capability, but again not explained.
* what drove their selection of USD rendering engine?  Which one was it?  If it didn't matter which one, I'd like to see that asserted.  
* what example AI/ML research workflow would this be a part of?

There are places where the paper's text flows poorly (below).  More importantly, while length limit can explain the persistence of my "unknowns" above, I think that (e.g.) USD could have been given more of an introduction at the sentence or paragraph level.  Also wanted to see some elaboration on the nature or quantity of the space/expressiveness "efficiency gains" that NS or USD representations offer.  Both of these things even within a space constraint.

If the paper had addressed those things, or had gone for the 8-page length and addressed items like my "remaining unknowns" more, and/or otherwise usefully lengthened the paper, I'd consider it an 'above threshold'.

Suggested text replacements (there might be another place or two where a pronoun doesn't match its referent in number):

Page 1:

* This lacks a verb:  "Recent advancements in 3D computer vision, including enhanced semantic segmentation (Barbosa and Os´orio 2023), robust object pose estimation techniques (Zhu et al. 2022), and the integration of 3D vision into autonomous systems (Singh and Bankiti 2023)."
* "Symbolic representations of data offer computational efficiency and storage benefits, making it an attractive choice for low-dimensional data. However, its limitations"
'representations' is plural, so should say 'making them an attractive choice' and 'their limitations'.
* "USD is embraced..."  USD is not defined here, so I can tell it has nice features, but not what it is.

Page 2:

* (BTW Section 2 title contains the string "Scens using USD".  It should have "Scenes".)

* "requires substituting part" should say "requires substituting parts".

* "related to the object in question" should say "related to the objects in question".

* "reconstruct symbolic representation of the object" should say "reconstruct symbolic representations of each object".

* "This research avenue holds the promise of seamlessly combining the advantages of compact 3D scene representation in this efficient data structure than point cloud."  The word "than" leads me to look for two things being compared, but I don't see that happening here.

* "These neuro-symbolic hybrid representations stored in USD formats..."  I think this is referring to data stored using the 'pointinstancer' schema.  The term 'hybrid' appears nowhere else in the paper--I'd like it to do so, or else not appear here.
title: title: Neuro-Symbolic Representations of 3D Scenes using Universal Scene Description Language
main_review: main_review: This work is well written though it could have presented more information on the approach. And could have done a better comparison against other similar approaches to show any sort of improvement over competing approaches. The question why is this approach better than the current state of the art was never quite answered. Just because you can do something doesn't mean you should do it. 

Neural Radiance Fields or other modern representations could also be mentioned for comparison, since point clouds have been improved upon. 

I don't know what their page count limit was, but the paper could be extended to six or eight pages to show the USD representation and the final neuro-symbolic representation of a single processed image, to further highlight what specifically is gained by taking this approach.  Also the reduction in space could be discussed, i.e., how much space is required by the original point cloud representation versus how much space is required by the final neuro-symbolic / USD representation. 

What was glaringly absent from this discussion was the amount of time taken to process a single image. No timing information was presented at all, which is concerning.  

The fact that a library of object images needs to be pre-constructed as well is also concerning. It is not mentioned where this library comes from. 

For the [near/far] future, it would be nice to generate the matched object library directly from the scene itself. When the object library becomes enormous, how costly will it be to match extant known objects against a new scene that is being processed in real-time, (24 frames per second).   Obviously the matching should be sub-second. 

Despite these shortcomings of the current paper, I think that this investigation is certainly noteworthy. 

The paper should be revised to address the shortcomings mentioned above.