title: Towards robust and generalisable natural language predicate inference
Creators: name: Li, Tianyi
Publisher: name: The University of Edinburgh
Year: 2025
Subjects: subject: Predicate Inference
subject: Entailment Graphs
subject: Directionality
subject: Language Models
subject: Cross-Lingual Complementarity

description: Natural language predicate inference is an important task of natural language processing, which bears particular significance in identifying supporting material for answering questions in the open domain. Given a premise and a hypothesis assertion, both concerning the same entities as arguments, natural language predicate inference aims to determine whether the hypothesis is entailed by the premise, i.e. whether the hypothesis can be concluded from the premise. In the task of predicate inference, directionality is a key characteristic, separating it from symmetric semantic similarity. Directionality is the property whereby the entailment relation between a pair of assertions holds in one direction but not both. To capture this property with an example, “John got his groceries at Tesco” entails “John went to Tesco”; however, “John went to Tesco” does not entail “John got his groceries at Tesco”. Directionality poses a unique challenge to predicate inference methods, especially those not making a sufficient distinction between entailment and similarity. Prior work has developed various approaches to natural language predicate inference. These approaches include: 1) Entailment graphs, a type of unsupervised symbolic methods, which are induced from predicate-argument mentions in vast natural language corpora; 2) fine-tuned Small Language Models, a type of supervised neural classifiers, trained on inference datasets; 3) few-shot generative interaction with Large Language Models, another unsupervised approach whose importance has been rapidly rising throughout the landscape of NLP and beyond. In this thesis, we explore the development of robust and generalisable predicate inference methods based on the above approaches. We start by generalizing entailment graphs to other languages than English. We demonstrate that strong entailment graphs can be built from Chinese corpora and reach strong predicate inference performance which is comparable to their English counterparts. Additionally, between the Chinese and English entailment graphs, we observe a mutual cross-lingual complementarity, where a simple ensemble of mono-lingual entailment graphs elicits substantially stronger performance than any individual graph. On the other hand, (small) language models, with their unified representation spaces for arbitrary text sequences, have become a common component in approaches to various classification tasks. We hypothesized that small language models can also be trained to perform predicate inference, and that multilingual language models would also benefit from the cross-lingual complementarity and exhibit improved performance when trained on multilingual inference datasets. However, through our analysis of fine-tuned small language models, we find little evidence of capability for the predicate inference task. Especially with directional entailments, fine-tuned language models overfit to dataset-specific artefacts that infest the inference datasets. On an extrinsic open-domain QA task, we observe that finetuned LMs exhibit weak predicate inference performance in both English and Chinese, where the performance is inferior to simple symbolic entailment graphs, despite the latter being sparse in coverage. With the rise of Large Language Models, which exhibit strong language understanding performance in general and few-shot learning capability, many NLP tasks are now considered solved by prompting LLMs with few-shot examples to generate the predictions. Since the few-shot in-context learning paradigm is independent of training datasets and the artefacts in them, we are able to use this setup to investigate the LLMs’ predicate inference capability. Through controlled experiments, we observe positive results overall, where few-shot LLMs yield non-trivial performance; on the other hand, we also observe worrying statistical biases in LLM answer generation. We identify prominent types of such biases, including an attestation bias and a relative frequency bias. We argue that these biases must be alleviated before LLMs can be trusted to perform robust natural language predicate inference in context. Overall, we develop and critically compare approaches to predicate inference, and advocate for a hybrid approach between efficient symbolic entailment graphs and versatile neural LLMs.