# Project SHADOW: Symbolic Higher-order Associative Deductive reasoning On Wikidata using LM probing

Hanna Abi Akl

► To cite this version:

Hanna Abi Akl. Project SHADOW: Symbolic Higher-order Associative Deductive reasoning On Wikidata using LM probing. NATL 2024 - 10th International Conference on Natural Language Computing, Nov 2024, Melbourne, Australia. hal-04706047

HAL Id: hal-04706047

https://hal.science/hal-04706047v1

Submitted on 24 Sep 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Project SHADOW: Symbolic Higher-order Associative Deductive reasoning On Wikidata using LM probing

Hanna Abi Akl

Data ScienceTech Institute, Université Côte d'Azur, Inria, CNRS, I3S, Paris, France

**Abstract.** We introduce SHADOW, a fine-tuned language model trained on an intermediate task using associative deductive reasoning, and measure its performance on a knowledge base construction task using Wikidata triple completion. We evaluate SHADOW on the LM-KBC 2024 challenge and show that it outperforms the baseline solution by 20% with a F1 score of 68.72%.

**Keywords:** Knowledge graphs, ontologies, natural language processing, large language models.

## 1 Introduction

Large language models (LLMs) have performed increasingly well in a wide range of semantic tasks including those involving leveraging knowledge from the models themselves [11]. This lead to research avenues investigating the capabilities of these models in knowledge-related tasks involving knowledge graphs and ontologies on the one hand, and measuring the intrinsic knowledge contained in LLMs on the other hand [11]. The Language Model Knowledge Base Construction (LM-KBC<fnref n="1" />) challenge proposes to evaluate intrinsic language model (LM) knowledge using techniques like LM probing and prompting [8] to construct knowledge bases by completing triples of subject entities and relations with the relevant object entities. In this work, we present SHADOW, or **Symbolic Higher-order Associative Deductive reasoning On Wikidata**, a fine-tuned model on knowledge base triples, and evaluate it on the LM-KBC task. We follow a methodology inspired from associative deductive reasoning [9] and leverage that technique to incorporate it in re-defining the probing problem to train the model more effectively. Specifically, we aim to address the following research questions: Can LLMs use transfer learning as a means to leverage their generative abilities to understand an association task? Can they use deduc-tive reasoning to solve it based on a neighboring task they have been previously trained on? The rest of the work is organized as follows. In section 2, we discuss some of the related work. Section 3 describes our experimental framework. In section 4, we report our results and discuss our findings. Finally, we conclude in section 5.

## 2 Related work

LM probing has been studied and evaluated in different research avenues. In their work, [10] study the information stored in LLMs with respect to their architecture, focusing on the factors behind their understanding of lexical semantics. Other techniques leverage prompting to encourage LLMs to use their knowledge more effectively to find better answers. [1] show that curating prompts manually and combining them with sets of entities

<footnote n="1">¹ https://lm-kbc.github.io/challenge2024/</footnote>
make relatively small LLMs perform well on knowledge base construction. Similar work
shows that LLMs can be prompted to generate seemingly coherent responses to incoherent
inputs [3].

Other lines of work treat LLMs as knowledge bases and employ query-based techniques to probe them for specific knowledge. [8] compare different transformer-based models by querying them with specific input knowledge and tracing whether this knowledge is retained in the LLMs. [2] evaluate LLMs as knowledge bases using different techniques ranging from model editing to discrete prompting on defined metrics like interpretability and causal tracing.

Research has also shown work on LLMs as knowledge bases using external knowledge in the form of extended vocabulary [6], knowledge sources like knowledge graph information [7] or by designing architectures that support external vectorized knowledge sources like Retrieval-Augmented-Generation (RAG) systems [5]. Finally, while deductive abilities for LLMs have been heavily investigated, [4] perform experiments to evaluate the inductive reasoning capabilities of these models and find that their deductive reasoning are much poorer than their inductive reasoning skills.

3 Experiments

This section describes our experiments in terms of data, model and training process.

3.1 Dataset

The data provided by the organizers are triples of the form (subject, relation, object). The
following relations are considered:

- **countryLandBordersCountry**: Null values possible (e.g., Iceland)

- personHasCityOfDeath: Null values possible

- seriesHasNumberOfEpisodes: Object is numeric

- awardWonBy: Many objects per subject (e.g., 224 Physics Nobel prize winners)

- companyTradesAtStockExchange: Null values possible

The data is provided in 3 sets: train, validation and test. The test set is used as the
official submission evaluation set. The number of triples in each set is:

- 377 in the train set

- 378 in the validation set

- 378 in the test set

For the subject and object in every triple, both the ID and the label are provided. A
sample triple is thus represented as such: {"SubjectEntity": "Belize", "SubjectEntityID":
"Q242", "ObjectEntities": ["Guatemala", "Mexico"], "ObjectEntitiesID": ["Q774", "Q96"],
"Relation": "countryLandBordersCountry"}.

3.2 Model

Formally, SHADOW is a generative model designed to solve the following function: y = f(x), where x is the input triple (s, p, ·) missing object entities, f is the function that generates a number in the set {1, 2, 3, 4, 5} and y is the corresponding template from the set of templates T = {t₁, t₂, t₃, t₄, t₅}, where t is a SPARQL query responsible for completing the knowledge graph triple with the missing objects. We train SHADOW as
a conditional generation model from a base flan-t5-small<fnref n="2" /> model and fine-tune it on the provided data. The training hyperparameters are configured as such:

- learning_rate: 1e-04

- train_batch_size: 4

- eval_batch_size: 4

- num_epochs: 20

- question_length: 512

- target_length: 512

- lr_scheduler_type: linear

- optimizer: Adam

- betas: 0.9, 0.999

- epsilon: 1e-08

## 3.3 Setup

We design our experiment to combine LLM probing with a symbolic component and indirectly evaluate the intrinsic knowledge found in LLMs on Wikidata knowledge graphs. We shift the focus away from generating correct SPARQL queries to retrieving the relevant objects for each subject and relation pair by designing templates containing the dynamic queries needed to answer the generic question:

*What Z completes the relationship Y for X?*,

where X, Y and Z refer respectively to the subject, relation and object(s) in a triple. Since the challenge deals with 5 types of relations, we design a total of 5 templates and assign a numerical template ID to each one of them. We then pair each subject and relation from a triple with the corresponding template ID which points to the correct SPARQL query that retrieves the corresponding object(s) to complete the triple based on the relation type. The SPARQL queries are themselves designed based on the Wikidata properties that best represent the targeted relation in the given triples.

SHADOW is then trained to generate the correct template ID depending on the given subject and relation without seeing the SPARQL queries. This requires the model learning on two fronts: first is by implicitly learning to associate the correct template with the relation type, and second by learning to generate a numerical value which corresponds to one of the acceptable template IDs defined in the frame of the experiment. Figure 1 shows our experimental design process.

The training is done by splitting the train set into an 80-20 split randomly and training on the 80%. The remaining 20% are incorporated into the validation set. We use the scikit-learn<fnref n="3" /> library to perform the data splits. Table 1 shows the training results.

The experiment is conducted on a Google Colab instance using a L4 High-RAM GPU. The code for our experimental setup is publicly available on GitHub<fnref n="4" />. The SHADOW model has been publicly released on Hugging Face<fnref n="5" />.

## 4 Results

Tables 2 and 3 capture the results on the official challenge test set. Overall, SHADOW performs well on the template identification task for the different relations. A closer inspection

<footnote n="2">² https://huggingface.co/google/flan-t5-small</footnote>

<footnote n="3">³ https://scikit-learn.org/stable/</footnote>

<footnote n="4">⁴ https://github.com/HannaAbiAkl/SHADOW</footnote>

<footnote n="5">⁵ https://huggingface.co/HannaAbiAkl/shadow</footnote>
<figure><img src="image_2.png" /><figcaption>Fig. 1. Experimental setup.</figcaption></figure>

<table><caption>**Table 1.** SHADOW model training results</caption><thead><tr><th>Training Loss</th><th>Epoch</th><th>Step</th><th>Validation Loss</th></tr></thead><tbody><tr><td>0.6679</td><td>1.0</td><td>1000</td><td>0.0005</td></tr><tr><td>0.3425</td><td>2.0</td><td>2000</td><td>0.0002</td></tr><tr><td>0.2303</td><td>3.0</td><td>3000</td><td>0.0001</td></tr><tr><td>0.1735</td><td>4.0</td><td>4000</td><td>0.000</td></tr><tr><td>0.1394</td><td>5.0</td><td>5000</td><td>0.0001</td></tr><tr><td>0.1167</td><td>6.0</td><td>6000</td><td>0.00009</td></tr><tr><td>0.1006</td><td>7.0</td><td>7000</td><td>0.00008</td></tr><tr><td>0.0882</td><td>8.0</td><td>8000</td><td>0.00007</td></tr><tr><td>0.0785</td><td>9.0</td><td>9000</td><td>0.00006</td></tr><tr><td>0.0707</td><td>10.0</td><td>10000</td><td>0.00006</td></tr><tr><td>0.0643</td><td>11.0</td><td>11000</td><td>0.00005</td></tr><tr><td>0.0590</td><td>12.0</td><td>12000</td><td>0.00005</td></tr><tr><td>0.0545</td><td>13.0</td><td>13000</td><td>0.00004</td></tr><tr><td>0.0506</td><td>14.0</td><td>14000</td><td>0.00004</td></tr><tr><td>0.0473</td><td>15.0</td><td>15000</td><td>0.00004</td></tr><tr><td>0.0443</td><td>16.0</td><td>16000</td><td>0.00003</td></tr><tr><td>0.0417</td><td>17.0</td><td>17000</td><td>0.00003</td></tr><tr><td>0.0394</td><td>18.0</td><td>18000</td><td>0.00003</td></tr><tr><td>0.0374</td><td>19.0</td><td>19000</td><td>0.00003</td></tr><tr><td>0.0355</td><td>20.0</td><td>20000</td><td>0.00003</td></tr></tbody></table>
of the results shows that the model performs worse on the *countryLandBordersCountry* relation which can be interpreted by the fact that the corresponding query targets property *P47*, or *shares border with*, which encompasses but is not limited to results sharing land borders (i.e. it also targets objects sharing sea borders). The nature of the property explains the high recall score, which retains a larger set of objects for that relation, and the low precision which reflects the few actual correct objects expected. The reason for choosing P47 is that it was the closest property that meets the required relation.

The result is validated in the zero-object cases, wherein the recall score is fairly high compared to a low precision score. This results is directly impacted by the choice of coding the queries as templates, sacrificing flexibility in results in favor of correct syntax instead of leaving the query generation in the hands of the model and risking some volatility in the results.

The relation *seriesHasNumberOfEpisodes* shows a contrasting result. The tradeoff between perfect precision and zero recall suggests a cautious classification, whereby the net of positive results, i.e. relations correctly associated with the proper template ID, is small but accurate. The precision score is explained by the fact that the query behind the template targets the correct property and returns the correct object (which is either a number or null if there are no series episodes). The low recall means SHADOW did not learn to associate this particular relation with its correct template, drawing questions over the intrinsic knowledge for that type of relation in the model. It is also possible that the stark difference between the nature of this relation, which is the only one among the 5 to expect a purely numerical answer as opposed to Wikidata entities, has proven more challenging for SHADOW to learn despite the fine-tuning process it has undergone.

Finally, it is worth highlighting the model's great performance on the *awardWonBy* relation, considering it is underrepresented at one-tenth of the other relations, i.e. 10 relations in the train, validation and test sets compared to approximately 100 for each of the four other relations. Since the model received less data for this relation compared to the others and still performed well, the performance could be explained by the nature of the internal knowledge the model has amassed for this relation, which in turn enables us to hypothesize on the good quality of the data given to this model at pre-training.

<table><caption>**Table 2.** Per-relation scores</caption><thead><tr><th>Relation</th><th>Precision</th><th>Recall</th><th>F1-score</th></tr></thead><tbody><tr><td>awardWonBy</td><td>0.9816</td><td>1.0000</td><td>0.9900</td></tr><tr><td>companyTradesAtStockExchange</td><td>0.9950</td><td>1.0000</td><td>0.9971</td></tr><tr><td>countryLandBordersCountry</td><td>0.7470</td><td>0.9717</td><td>0.7829</td></tr><tr><td>personHasCityOfDeath</td><td>0.9700</td><td>1.0000</td><td>0.9700</td></tr><tr><td>seriesHasNumberOfEpisodes</td><td>1.0000</td><td>0.0000</td><td>0.0000</td></tr><tr><td>Average</td><td>0.9453</td><td>0.7297</td><td>0.6872</td></tr></tbody></table>

<table><caption>**Table 3.** Zero-object cases</caption><thead><tr><th>Precision</th><th>Recall</th><th>F1-score</th></tr></thead><tbody><tr><td>0.4975</td><td>0.90006</td><td>0.6408</td></tr></tbody></table>
Table 4 shows the official performance of SHADOW compared to other solutions. Despite its difficulties with some relations, SHADOW performs very well with respect to the baseline and outperforms it by almost 20%. It falls however short of the best scores and is outclassed by some margin.

<table><caption>**Table 4.** Official submission leaderboard</caption><thead><tr><th>Team Name</th><th>Average F1-score</th></tr></thead><tbody><tr><td>davidebara</td><td>0.9224</td></tr><tr><td>KB</td><td>0.9131</td></tr><tr><td>RAGN4ROKS</td><td>0.9083</td></tr><tr><td>WWWD</td><td>0.6977</td></tr><tr><td><strong>DSTI</strong></td><td><strong>0.6872</strong></td></tr><tr><td>NadeenFathallah</td><td>0.6529</td></tr><tr><td>Rajaa</td><td>0.5662</td></tr><tr><td>aunsiels</td><td>0.5076</td></tr><tr><td>lm-kbc-organizer</td><td>0.4865</td></tr></tbody></table>

# 5 Conclusion

In this work, we show how a fine-tuned LLM model can leverage intrinsic knowledge through LM probing and combine it with associative deductive reasoning to build dis-ambiguated knowledge bases. The performance of SHADOW, our model, outperforms the baseline by disambiguating relation types and indirectly associating them with rele-vant knowledge graph completion queries. Our experiments show however that LLMs are highly influenced by the type of data they have already been trained on and possess uneven knowledge with respect to Wikidata relations which leaves much room for improvement in that area. Future work will focus on studying relation types in depth to improve LLM knowledge and using that knowledge to further evaluate LLM reasoning capabilities.

# References

1. Alivanistos, Dimitrios and Santamaría, Selene Báez and Cochez, Michael and Kalo, Jan-Christoph and van Krieken, Emile and Thanapalasingam, Thiviyan: Prompting as probing: Using language models for knowledge base construction. arXiv preprint arXiv:2208.11057 (2022)

2. AlKhamissi, Badr and Li, Millicent and Celikyilmaz, Asli and Diab, Mona and Ghazvininejad, Marjan: A review on language models as knowledge bases. arXiv preprint arXiv:2204.06031 (2022)

3. Cherepanova, Valeriia and Zou, James: Talking Nonsense: Probing Large Language Models' Understanding of Adversarial Gibberish Inputs. arXiv preprint arXiv:2404.17120 (2024)

4. Cheng, Kewei and Yang, Jingfeng and Jiang, Haoming and Wang, Zhengyang and Huang, Binxuan and Li, Ruirui and Li, Shiyang and Li, Zheng and Gao, Yifan and Li, Xian and others: Inductive or deductive? Rethinking the fundamental reasoning abilities of LLMs. arXiv preprint arXiv:2408.00114 (2024)

5. Li, Belinda Z and Liu, Emmy and Ross, Alexis and Zeitoun, Abbas and Neubig, Graham and Andreas, Jacob: Language Modeling with Editable External Knowledge. arXiv preprint arXiv:2406.11830 (2024)

6. Loureiro, Daniel and Jorge, Alípio Mário: Probing Commonsense Knowledge in Pre-trained Language Models with Sense-level Precision and Expanded Vocabulary. arXiv preprint arXiv:2210.06376 (2022)

7. He, Qiyuan and Wang, Yizhong and Wang, Wenya: Can Language Models Act as Knowledge Bases at Scale?. arXiv preprint arXiv:2402.14273 (2024)
8. Petroni, Fabio and Rocktäschel, Tim and Riedel, Sebastian and Lewis, Patrick and Bakhtin, Anton and Wu, Yuxiang and Miller, Alexander: Language Models as Knowledge Bases?. *Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)*, pp 2463–2473 (2019)

9. Takano, Wataru and Nakamura, Yoshihiko: Associative processes between behavioral symbols and a large scale language model. *2010 IEEE International Conference on Robotics and Automation*, pp 2404–2409 (2010)

10. Vulić, Ivan and Ponti, Edoardo Maria and Litschko, Robert and Glavaš, Goran and Korhonen, Anna: Probing pretrained language models for lexical semantics. *Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)*, pp 7222–7240 (2020)

11. Wang, Xin and Chen, Zirui and Wang, Haofen and Li, Zhao and Guo, Wenbin and others: Large Language Model Enhanced Knowledge Representation Learning: A Survey. arXiv preprint arXiv:2407.00936 (2024)

## Authors

**Hanna Abi Akl** is a professor and researcher in logic, language, computation and neuro-symbolic artificial intelligence. He holds a Masters degree in Computer Science from the Université Côte d'Azur and an MSc in Data Science and Artificial Intelligence from Data ScienceTech Institute. His main areas of research are mapping language to symbolic structures, logic-based reasoning and graph-based knowledge representation methods. Hanna works as a researcher at INRIA and holds the Vice-Dean position at Data ScienceTech Institute.