Question: Does their embedding space preserve the semantic properties of the conceptual elements that make up visual scenes?
Evidence: Recently, there has been a significant increase in the investment for autonomous driving (AD) research and development, with the goal of achieving full autonomy in the next few years. Realizing this vision requires robust ML/AI algorithms that are trained on massive amounts of data. Thousands of cars, equipped with various types of sensors (e.g., LIDAR, RGB, RADAR), are now deployed around the world to collect this heterogeneous data from real-world driving scenes. The primary objective for AD is to use these data to optimize the vehicle's perception pipeline on such tasks as: 3D object detection, obstacle tracking, object trajectory forecasting, and learning an ideal driving policy. Fundamental to all of these tasks will be the vehicle's context understanding capability, which requires knowledge of the time, location, detected objects, participating events, weather, and various other aspects of a driving scene. Even though state-of-the-art AI technologies are used for this purpose, their current effectiveness and scalability are insufficient to achieve full autonomy. Humans naturally exhibit context understanding behind the wheel, where the decisions we make are the result of a continuous evaluation of perceptual cues combined with background knowledge. For instance, human drivers generally know which area of a neighborhood might have icy road conditions on a frigid winter day, where flooding is more frequent after a heavy rainfall, which streets are more likely to have kids playing after school, and which intersections have poor lighting. Currently, this type of common knowledge is not being used to assist self-driving cars and, due to the sample-inefficiency of current ML/AI algorithms, vehicle models cannot effectively learn these phenomena through statistical observation alone. On March 18, 2018, Elaine Herzberg’s death was reported as the first fatality incurred from a collision with an autonomous vehicle. An investigation into the collision, conducted by The National Transportation Safety Board (NTSB), remarks on the shortcomings of current AD and context understanding technologies. Specifically, NTSB found that the autonomous vehicle incorrectly classified Herzberg as an unknown object, a vehicle, and then a bicycle within the complex scene as she walked across the road. Further investigation revealed that the system design did not include consideration for pedestrians walking outside of a crosswalk, or jaywalking BIBREF9. Simply put, the current AD technology lacks fundamental understanding of the characteristics of objects and events within common scenes; this suggests that more research is required in order to achieve the vision of autonomous driving.
Knowledge Graphs (KGs) have been successfully used to manage heterogeneous data within various domains. They are able to integrate and structure data and metadata from multiple modalities into a unified semantic representation, encoded as a graph. More recently, KGs are being translated into latent vector space representations, known as Knowledge Graph Embeddings (KGEs), that have been shown to improve the performance of machine learning models when applied to certain downstream tasks, such as classification BIBREF10, BIBREF11. Given a KG as a set of triples, KGE algorithms learn to create a latent representation of the KG entities and relations as continuous KGE vectors. This encoding allows KGEs to be easily manipulated and integrated with machine learning algorithms. Motivated by the shortcomings of current context understanding technologies, along with the promising outcomes of KGEs, our research focuses on the generation and evaluation of KGEs on AD data. Before directly applying KGEs on critical AD applications, however, we evaluate the intrinsic quality of KGEs across multiple metrics and KGE algorithms BIBREF12. Additionally, we present an early investigation of using KGEs for a selected use-case from the AD domain.
Dataset. To promote and enable further research on autonomous driving, several benchmark datasets have been made publicly available by companies in this domain BIBREF13. NuScenes is a benchmark dataset of multimodal vehicular data, recently released by Aptiv BIBREF14 and used for our experiments. NuScenes consists of a collection of 20-second driving scenes, with $\sim $40 sub-scenes sampled per driving scene (i.e., one every 0.5 seconds). In total, NuScenes includes 850 driving scenes and 34,149 sub-scenes. Each sub-scene is annotated with detected objects and events, each defined within a taxonomy of 23 object/event categories.
Scene Ontology. In autonomous driving, a scene is defined as an observable volume of time and space BIBREF15. On the road, a vehicle may encounter many different situations|such as merging onto a divided highway, stopping at a traffic light, and overtaking another vehicle|all of which are considered as common driving scenes. A scene encapsulates all relevant information about a particular situation, including data from vehicular sensors, objects, events, time and location. A scene can also be divided into a sequence of sub-scenes. As an example, a 20-second drive consisting primarily of the vehicle merging into a highway could be considered as a scene. In addition, all the different situations the vehicle encounters within these 20 seconds can also be represented as (sub-)scenes. In this case, a scene may be associated with a time interval and spatial region while a sub-scene may be associated with a specific timestamp and a set of spatial coordinates. This semantic representation of a scene is formally defined in the Scene Ontology (see figure FIGREF8(a), depicted in Protege). To enable the generation of a KG from the data within NuScenes, the Scene Ontology is extended to include all the concepts (i.e., objects and event categories) found in the NuScenes dataset.
Generating Knowledge Graphs. The Scene Ontology identifies events and features-of-interests (FoIs) as top-level concepts. An event or a FoI may be associated with a Scene via the includes relation. FoIs are associated with events through the isParticipantOf relation. Figure FIGREF8(b) shows a subset of the FoIs and events defined by the Scene Ontology. In generating the scenes' KG, each scene and sub-scene found in NuScenes is annotated using the Scene Ontology. Table TABREF9 shows some basic statistics of the generated KG.
KGE Algorithms. KGE algorithms enable the ability to easily feed knowledge into ML algorithms and improve the performance of learning tasks, by translating the knowledge contained in knowledge graphs into latent vector space representation of KGEs BIBREF16. To select candidate KGE algorithms for our evaluation, we referred to the classification of KGE algorithms provided by Wang et al. BIBREF17. In this work, KGE algorithms are classified into two primary categories: (1) Transitional distance-based algorithms and (2) Semantic matching-based models. Transitional distance-based algorithms define the scoring function of the model as a distance-based measure, while semantic matching-based algorithms define it as a similarity measure. Here, entity and relation vectors interact via addition and subtraction in the case of Transitional distance-based models; in semantic matching-based models, the interaction between entity and relation vectors is captured by multiplicative score functions BIBREF18.
Initially, for our study we had selected one algorithm from each class: TransE BIBREF19 to represent the transitional distance-based algorithms and RESCAL BIBREF20 to represent the semantic matching-based algorithms. However, after experimentation, RESCAL did not scale well for handling large KGs in our experiments. Therefore, we also included HolE BIBREF21|an efficient successor of RESCAL|in the evaluation. A brief summary of each algorithm is provided for each model, below:
TransE: the TransE model is often considered to be the most-representative of the class of transitional distance-based algorithms BIBREF17. Given a triple (h, r, t) from the KG, TransE encodes h, r and t as vectors, with r represented as a transition vector from h to t: $\mathbf {h} + \mathbf {r} \approx \mathbf {t}$. Since both entities and relations are represented as vectors, TransE is one of the most efficient KGE algorithms, with $\mathcal {O}(n d + m d)$ space complexity and $\mathcal {O}(n_t d)$ time complexity ($n_t$ is the number of training triples).
RESCAL: RESCAL is capable of generating an expressive knowledge graph embedding space, due to its ability to capture complex patterns over multiple hops in the KG. RESCAL encodes relations as matrices and captures the interaction between entities and relations using a bi-linear scoring function. Though the use of a matrix to encode each relation yields improved expressivity, it also limits RESCAL’s ability to scale with large KGs. It has $\mathcal {O}(n d + m d^2)$ space complexity and $\mathcal {O}(n_t d^2)$ time complexity.
HolE: HoLE is a more efficient successor of RESCAL, addressing its space and time complexity issues, by encoding relations as vectors without sacrificing the expressivity of the model. By using circular correlation operation BIBREF21, it captures the pairwise interaction of entities as composable vectors. This optimization yields $\mathcal {O} (n d + m d)$ space complexity and $\mathcal {O}(n_t d \log d)$ time complexity.
Visualizing KGEs. In order to visualize the generated KGE, a “mini" KG from the NuScenes-mini dataset was created. Specifically, 10 scenes were selected (along with their sub-scenes) to generate the KG, and the TransE algorithm was used to learn the embeddings. When training the KGEs, we chose the dimension of the vectors to be 100. To visualize the embeddings in 2-dimensional (2D) space, the dimensions are reduced using the t-Distributed Stochastic Neighbor Embedding (t-SNE) BIBREF22 projection. Figure FIGREF11(a) shows the resulting embeddings of the NuScenes dataset. To denote interesting patterns that manifest in the embeddings, instances of Car (a FoI) and the events in which they participate are highlighted. In this image, events such as parked car, moving car, and stopped car are clustered around entities of type Car. This shows that the isParticipantOf relations defined in the KG are maintained within the KG embeddings.

Here, we deviate slightly from the prior work in evaluating KGE algorithms, which evaluate KGEs based downstream task performance. Instead, we focus on an evaluation that uses only metrics that quantify the intrinsic quality of KGEs BIBREF23: categorization measure, coherence measure, and semantic transition distance. Categorization measures how well instances of the same type cluster together. To quantify this quality, all vectors of the same type are averaged together and the cosine similarity is computed between the averaged vector and the typed class. The Coherence measure quantifies the proportion of neighboring entities that are of the same type; the evaluation framework proposes that, if a set of entities are typed by the class, those entities should form a cluster in the embedding space with the typed class as the centroid. Adapted from the word embedding literature, Semantic Transitional Distance captures the relational semantics of the KGE: if a triple $(h,r,t)$ is correctly represented in the embedding space, the transition distance between the vectors representing $(\mathbf {h+r})$ should be close to $\mathbf {t}$. This is quantified by computing the cosine similarity between $(\mathbf {h+r})$ and $\mathbf {t}$.
Results. Evaluation results are reported with respect to each algorithm and metric. Figure FIGREF13 shows the evaluation results of categorization measure, coherence measure, and semantic transitional distance|for each KGE algorithm. The NuScenes KG, generated from the NuScenes-trainval dataset, is large in terms of both the number of triples and number of entities (see Table TABREF9). Hence, RESCAL did not scale well to this dataset. For this reason, we only report the evaluation results for TransE and HolE. When considering the KGE algorithms, TransE's performance is consistently better across metrics, compared to HolE's performance. However, it is interesting to note that HolE significantly outperforms TransE for some classes/relations. When considering the evaluation metrics, it is evident that the categorization measure and semantic transitional distance are able to capture the quality of type semantics and relational semantics, respectively. The value of the coherence measure, however, is zero for HoLE in most cases and close to zero for TransE in some cases. In our experimental setting, the poor performance with respect to the coherence measure may suggest that it may not be a good metric for evaluating KGEs in the AD domain.
We report preliminary results from our investigation into using KGEs for a use-case in the AD domain. More specifically, we apply KGEs for computing scene similarity. In this case, the goal is to find (sub-)scenes that are characteristically similar, using the learned KGEs. Given a set of scene pairs, we choose the pair with the highest cosine similarity as the most similar. Figure FIGREF15 shows an illustration of the two most similar sub-scenes, when the list of pairs include sub-scenes from different scenes. An interesting observation is that the black string of objects in sub-scene (a) are Barriers (a Static Object), and the orange string of objects in sub-scene (b) are Stopped Cars. This example suggests that the KGE-based approach could identify sub-scenes that share similar characteristics even though the sub-scenes are visually dissimilar.
We presented an investigation of using KGEs for AD context understanding, along with an evaluation of the intrinsic quality of KGEs. The evaluation suggests that KGEs are specifically able to capture the semantic properties of a scene knowledge graph (e.g., isParticipantOf relation between objects and events). More generally, KGE algorithms are capable of translating semantic knowledge, such as type and relational semantics to KGEs. When considering the different KGE algorithms, we report that the transitional distance-based algorithm, TransE, shows consistent performance across multiple quantitative KGE-quality metrics. Our evaluation further suggests that some quality metrics currently in use, such as the coherence measure, may not be effective in measuring the quality of the type semantics from KGEs, in the AD domain. Finally, in applying the learned KGEs to a use-case of importance in the AD domain, we shed some light on the effectiveness of leveraging KGEs in capturing AD scene similarity.
We suggest neuro-symbolic fusion as one solution and, indeed, our results show that our embedding space preserves the semantic properties of the conceptual elements that make up visual scenes.
Answer: Yes.