Dimensional Embedding Techniques for Knowledge Graph Mapping and LLMs

Authors: liubov  tupikina
Venue: KDD_org_2024_Workshop_DL4KG
Published: 2024-06-10 23:19:01

Abstract: We propose a framework, which aims at construction of explainable data embedding methods. It is specifically based on the low-dimensional embedding techniques which are connecting higher-order geometric analysis, topological data analysis and natural language processing methods.
We consider the applications of our framework to the navigation of the knowledge scape is non-trivial in the everyday context, when knowledge/data growth is beyond exponential. Moreover our framework supports methods for generating knowledge graph (node) embeddings, and temporal knowledge graph embeddings.

Main paper: 
 introduction: 
 In the era of information abundance, navigating the knowledge scape and utilizing large language models (LLMs) trained on vast amounts of textual data has become a significant challenge [1] [2] [3] . The exponential growth of data sources necessitates the development of robust and explainable dimensionality reduction techniques to process this high-dimensional data effectively.Despite their remarkable performance in various natural language processing tasks, large language models (LLMs) [3, 6] are trained on general-purpose data and have lower performance in domain-specific tasks. Additionally, LLMs suffer from hallucination problems, and they are opaque models that lack interpretability. A potential solution to these problems is to induce the knowledge from knowledge graphs to LLMs [6] . TDA studies the shape by going beyond the standard measures defined on data points pairs. We move from networks with simple edges to simplicial complexes. Simplicial complexes are obtained from elementary objects called simplices. Simplices can be built from simple points, line segments, triangles, and other higher order structures all glued together. This representation in latent space provides more interpretable results to visualize relationships in data. To uncover hidden patterns and relationships in data that may not be apparent through traditional analysis methods, we explore higher-order data analysis [8] . By considering higher-order interactions, we aim to gain deeper insights into complex datasets. This paper proposes the use of low-dimensional embedding techniques for working with LLMs at various stages, including training and analysis of latent space. These techniques are crucial for understanding and managing the complex relationships between entities in the knowledge graph, which is essential for effective knowledge representation and reasoning [6] .The need for such techniques arises from the fact that LLMs are trained on massive datasets, making it difficult to interpret their behavior and decisions. By reducing the dimensionality of the data, we can gain insights into the underlying patterns and structures that govern the behavior of these models. In the contemporary era, where knowledge and data growth is beyond exponential, navigating the knowledge scape becomes a non-trivial task. To address this challenge, in this methodological survey paper we propose to use low-dimensional embedding techniques connected to knowledge graph mapping and grounded in the theory of topological data analysis and higher-order structures. This approach aims to provide a more comprehensive understanding of complex relationships within high-dimensional datasets and specifically in the context of designing explainable frameworks for LLMs. In the Methods section we explain about the main methodological framework of topology infused embeddings. In the Results and Discussions section we describe the main methodological advances and possible perspectives.Our proposed approach for explainable mathematical foundations for AI systems models, such as LLMs, is based on the idea of combination of neural network latent space investigation using hypergraph approaches and dimensionality reduction techniques. For demonstrating our approach we use several embedding models (BERT, general autoencoder architectures) [3] applying them to the high-dimensional textual data with some ground-truth information about the datasets. By ground-truth we mean the information about the interconnection in data. The main idea behind the method is that then using our approach the results of the embeddings to texts then can be studied as geometrical structures, such as manifolds or hypergraphs (Figure 1 ), which generally simplifies and creates yet another language for analysis of LLMs and evaluation of LLMs.The method of topology infused embeddings consists of several main stages:• Data preprocessing: we preprocess the data by cleaning and normalizing it to improve the quality of the input.• Embedding models, when we use embedding models, such as BERT or general autoencoder architectures, to transform the high-dimensional textual data into low-dimensional embeddings. These embeddings capture the semantic relationships between words and phrases in the text.• Next, we study the neural network latent space using higher-order data analysis (hypergraph theory) and geometry infused topological properties of such structures.We analyze the embeddings using topological methods, such as manifold learning or hypergraph analysis. These methods allow us to identify patterns and structures in the data that are not visible in the original high-dimensional space. Then we interpret the results of the topological analysis to understand the underlying semantics of the data. For example, we can identify clusters of similar words or phrases, or we can visualize the relationships between different concepts.The topological properties of some of the regions of such embeddings then can be directly linked and related to the projected content from texts, which helps us to relate the higher-order information (in this context, it is text) with some geometrical and topological properties of such learned embedded objects (such as manifolds). Overall we believe that this brings us to the next level of data analysis, where neural networks architectures of embedding methods can be linked with topological data analysis dimensionality reduction techniques [4, 5] . Hence this may be a pathway towards resolving some aspects of explainability issues in LLMs or over-reliance on training data in LLMs, since with such an approach we aim to find some invariant correspondence between textual and non-textual structures. The system may not have been trained directly on the benchmark, but on similar items that necessitate comparable reasoning patterns. We extend our method by investigating some of the benchmarks and knowledge databases. Moreover we demonstrate its applications in mapping of news ideas that appear through time and how knowledge grows [1] , as well as generally map and represent high dimensional (textual) data in the lower dimensional representation with the explainable AI methods. In one of the previous applications of our method in previous research papers [1] we looked at millions of articles in science, to see how the knowledge landscape evolves [1, 2] . One of the key advantages of higher-order interaction analysis is its ability to capture non-linear and non-monotonic relationships between variables. This allows us to model more complex phenomena and make more accurate predictions. For example, higher-order interactions could reveal how the combination of different demographic factors (e.g., age, gender, income) influences consumer behavior, or in the context of scientometrics project [1, 2] it can be simply a combination of new scientific fields emerging.The approach combines neural network latent space investigation using hypergraph approaches and dimensionality reduction techniques, which allows us to study the results of embeddings as geometrical structures. This simplifies and creates yet another language for analysis of LLMs, making it easier to relate higher-order information (text) with geometrical and topological properties of embedded objects.This method can be used to explore the potential for using LLMs as tools for knowledge discovery and information retrieval. By mapping news ideas that appear over time, we can see how knowledge grows and evolves. This approach can also be applied to other types of high-dimensional data, such as scientific articles or social media posts.In conclusion, the proposed method is a promising approach for explainable mathematical foundations for AI systems models, such as LLMs. It provides a framework for understanding how LLMs process and generate text, and it can be used for exploring the potential of LLMs for knowledge discovery and information retrieval.* Corresponding author. liubov.tupikina@nokia-bell-labs.com ORCID number 0000-0002-7169-5706 ©Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). 
 
 Appendix:  



Reviews: review: title: title: Unstructured with no clear contribution.
main_review: main_review: The paper superficially describes the idea of using topological methods in analyzing and interpreting LLMs.

# Weak Points
- Technical details are missing. Scientific contribution is vague.
- Texts are repetitive.
- Methodology is lacking in detail.
- Proposed method is similar to (Park et al., 2024) https://arxiv.org/pdf/2406.01506
- The CEUR-WS LaTeX template was not used.

# Minor Comments:
- Page 1: What is TDA?
- Reference #8: Perhaps cite the paper instead of a collab notebook?

# Score
## Quality (1-5): 1 (very poor)
*Unstructured, lacking in literature review."

## Clarity (1-5): 1 (very poor)
*Proposed approach not clearly described or motivated."*

## Originality (1-5): 1 (marginal)

## Relevance to the Conference (1-5): 2 (only marginal relevance)
*The combination of graph theory and LLMs are relevant to this workshop. However, sans scientific contribution deems this relevance moot."

## Impact (1-5): 1 (very low)
*No clear contribution."

## Technical Soundness (1-5): 1 (very low)
*Lacking in technical details.*
title: title: Interesting research direction but methodological details unclear
main_review: main_review: Dear author,

I find it very relevant to carry out research with the goal of making LLMs more understandable and combining LLMs with knowledge graphs. Reading your paper gives me the impression that you have plans to carry out research in this direction, but it seems like you have not yet worked out the details of the methods nor have you carried out experiments to validate your beliefs.

There a a couple of specific things that were not clear to me or that I'd like to mention:

1. "robust and explainable dimensionality reduction techniques". It seems like you regard textual data as being high-dimensional. Then, does dimensionality reduction mean that you train an LLM, for example to obtain vectors in, e.g., 300 dimensional space?

2. LLMs are trained on general-purpose data and have lower performance in domain-specific tasks. This is true, but these models can be fine-tunes for specific tasks.

3. You use the acronym "TDA" but you never explain what it is. Only later I understood that this probably is topological data analysis.

4. "This representation in latent space provides more interpretable results to visualize relationships in data". I do not understand what these representations are and what the role of simplicial complexes are. But aren't vector spaces in general difficult to understand?

5. The method is entirely unclear to me. You mention that you combine dimensionality reduction (which in your case means training an LLM) with knowledge graph mapping (I am unsure what that means. Is this about embedding entities and relations from a knowledge graph?), somehow making use of methods from topological data analysis (where it is not clear to me which methods, what they do, how they are combined). What really are topologically-infused embeddings? Is a KG relevant here? Figure 1 does not help me to understand the method.

6. I have the impression that the paper is not sufficiently self-contained. One should be able to understand the main idea without reading [1] and [2].

7. You did not use the official template, but I reviewed your paper anyways.

In general, as I wrote in the beginning, this is an interesting research direction. But it might be in a too early state for a publication. My recommendation is to carve out a research question or hypothesis for which data exists or can be created so that you can carry out an experiment to gain insights. Then develop the minimal core of your method and carry out an experiment. Maybe advance step by step. Maybe topology-infusion is already sufficiently interesting - no need for external knowledge from knowledge graphs (unless I misunderstand you and one needs external knowledge for topology infusion).
title: title: Insuffiently precise to be of interest.
main_review: main_review: This paper appears to present a proposed research framework for the embedding of knowledge. There are no results or evaluations. A position paper on the development of such a framework, presented before the work is done, may be an interesting contribution to a workshop. However, in such a paper the following should at least be present:

1) A clear definition of the proposed framework, including clear definitions of of terms (such as "higher-order interactions", "knowledge scape" and "topology infused embeddings"). At the moment, the framework is not sufficiently precisely described for me to form any judgment.
2) An extensive embedding of the idea in the related work. In this case, there is a huge amount of literature on geometric learning using, for example, non-Euclidean embeddings, group theory or Clifford algebras, all of which may be relevant. The paper should clearly indicate these jumping off points and the novelty of the proposed approach.
3) Clear, falsifiable predictions. The paper should make clear predictions for what the proposed research program will achieve. These predictions should be stated clearly enough to be evaluated, and be precise enough to be falsifiable.

Since the paper fails to provide any of these, I can only recommend a strong reject. I expect that the idea has merit, and I hope the author will pursue this line of research, but it should be much further developed before being submitted for review.