Supposedly Equivalent Facts That Aren’t? Entity Frequency in Pre-training Induces Asymmetry in LLMs
Authors: Yuan He
Bailan He
Zifeng Ding
Alisia Maria Lupidi
Yuqicheng Zhu
Shuo Chen
Caiqi Zhang
Jiaoyan Chen
Yunpu Ma
Volker Tresp
Ian Horrocks
Venue: COLM 2025 (2025)
Keywords: Large Language Models
Asymmetry
Equivalent Facts
Entity Frequency
Pre-training Bias
Knowledge Probing
Hallucinations
Knowledge Graphs
TL;DR: This work demonstrates that the asymmetry in how large language models recognise equivalent facts stems from inherent biases in their pre-training data, particularly through differences in entity frequency.

Abstract: Understanding and mitigating hallucinations in Large Language Models (LLMs) is crucial for ensuring reliable content generation. While previous research has primarily focused on "when" LLMs hallucinate, our work explains "why" and directly links model behaviour to the pre-training data that forms their prior knowledge. Specifically, we demonstrate that an asymmetry exists in the recognition of logically equivalent facts, which can be attributed to frequency discrepancies of entities appearing as subjects versus objects. Given that most pre-training datasets are inaccessible, we leverage the fully open-source $\texttt{OLMo}$ series by indexing its $\texttt{Dolma}$ dataset to estimate entity frequencies. Using relational facts (represented as triples) from $\texttt{Wikidata5M}$, we construct probing datasets to isolate this effect. Our experiments reveal that facts with a high-frequency subject and a low-frequency object are better recognised than their inverse, despite their logical equivalence. The pattern reverses in low-to-high frequency settings, and no statistically significant asymmetry emerges when both entities are high-frequency. These findings highlight the influential role of pre-training data in shaping model predictions and provide insights for inferring the characteristics of pre-training data in closed or partially closed LLMs.

Published as a conference paper at COLM 2025
Supposedly Equivalent Facts That Aren't? Entity Frequency in Pre-training Induces Asymmetry in LLMs
Yuan Hel.2+t, Bailan H Zifeng Dingå Alisia Lupidi Yuqicheng Zhu?
Shuo Chen², Caiqi Zhang Jiaoyan Chen', Yunpu Ma Volker Tresp Ian Horrocks² 1 Amazon, AUniversity of Oxford, LLM Munich, *Siemens AG, UUniversity of Cambridge, Meta, University of Stuttgart, Bosch Center for AI, Thhe University of Manchester, Munich Center for Machine Learning lawhy@amazon.com, 7d320@cam.ac.uk
Abstract
Understanding and mitigating hallucinations in Large Language Models (LLMs) is crucial for ensuring reliable content generation. While previous research has primarily focused on "when" LLMs hallucinate, our work explains "why" and directly links model behaviour to the pre-training data that forms their prior knowledge. Specifically, we demonstrate that an asymmetry exists in the recognition of logically equivalent facts, which can be attributed to frequency discrepancies of entities appearing as subjects versus objects. Given that most pre-training datasets are inaccessible, we leverage the fully open-source OLMo series by indexing its Dolma dataset to estimate entity frequencies. Using relational facts (represented from Wikidata5M, we construct probing datasets to isolate this effect. Our experiments reveal that facts with high-frequency subject and low- frequency object are better recognised than their inverse, despite their logical equivalence. The pattern reverses in low-to-high frequency settings, and no statistically significant asymmetry emerges when both entities are high-frequency. These findings highlight the influential role of pre-training data in shaping model predictions and provide insights for inferring the characteristics of pre-training data in closed partially closed LLMs.
triples)
1 Introduction
Large Language Models (LLMs) have demonstrated remarkable success in generating fluent and contextually relevant text (Brown et al., 2020; Achiam et al., 2023; Anil et al., 2023; Dubey et al., 2024). However, their tendency to produce hallucinated inconsistent information remains critical challenge (Huang et al., 2025; Rawte et al., 2023), particularly as these models are increasingly deployed in applications where reliability is paramount (Liu et al., 2023; Huang et al., 2024).
factually
Traditionally, research has focused on identifying the circumstances under which halluci- nations occur. For example, Lin et al. (2022) found that LLMs struggle with truthfulness when confronted with conspiracy-style prompts, while Lin et al. (2024) demonstrated their vulnerability to variations in language style through question paraphrasing, Additionally, Berglund et al. (2023) highlighted a structural limitation by showing that LLMs often fail to infer reverse implications correctly when fine-tuned on synthetic forward implications. In contrast to these approaches, our work addresses a more fundamental question: How does the pre-training data the very source of an L.M's prior knowledge influence its propensity to hallucinate?
* "quual contribution.
t Work done prior to joining Amazon.
'See code GitHub: https: thwb. com//KKR--Oxford/Factttooo and datasets on Zenodo: https://doi.org/// 5281 5281/zenodo. 15892788.

---

Published as a conference paper at COLM 2025
We posit that one key factor lies in the frequency distribution of entities in the pre-training corpus, As illustrated in Figure 1, an LLM may correctly recognise that the football star Diego Maradona has a sibling named Raul Maradona (a lesser-known individual), yet struggle with the inverse recognition that Raul Maradona has a sibling named Diego Maradona, despite both statements conveying the same fact. Our central hypothesis is that discrepancies in enfity frequencies during pre-training introduce bias into the model's 's predictive distribution over the correctness of equivalent facts. We aim to analyse and quantify this phenomenon to gain new insights into how pre-training data influence factual reliability in LLMs.
Given that most pre-training datasets are proprietary or otherwise inaccessible (Shi et al., 2023), we leverage the fully open- source OLMo series (Groeneveld et al., 2024; OL Team et al., 2024). By indexing its ac- companying Dolma pre-training dataset (Sol- daini et al., 2024), we are able to estimate en- tity frequencies reliably. To isolate the effect of entity frequencies, we construct probing datasets from relational facts extracted from Wikidata5M, represented as triples (s, r, o)- Our experiments reveal consistent pattern: facts formatted as (s, r, o), where the sub- ject s is high-frequency and the object o is low-frequency, are more readily recognised than their logically equivalent inversions, (o, وا,,, s); when the frequency dynamics are reversed (i.e., low-frequency subject and high-frequency object), the recognition pattern flips; when both entities are
Free.
Sibling
Diego Maradona
Low Freq.
Sibling
Pre-training Freq.
Raul
Figure 1: LLMs can exhibit asymmetry when recognising equivalent facts, often iden- tifying facts from high-frequency to low- frequency entities but struggling with the in- verse. Shown here is a working example from our tests with the OLMo2-13B 13B model.
the asymmetry is neither prominent nor statisti- cally significant. These findings offer novel insights into an under-explored aspect of model behaviour tied directly to pre-training data, and they motivate further research for inferring characteristics of pre-training corpora- especially for models with undisclosed pre-training datasets.
2 Related Work
The term "reversal curse" was first introduced by Berglund et al. (2023) to describe the structural inability of auto-regressive LLMs to infer "B entailed by A" when trained on "A entails B". Their work demonstrated this phenomenon using fine- tuned models trained on synthetically crafted datasets, such as mameestoo-dsscrition and questions-to toattions-tt-nsweer Further analysis by Zhu et al. (2024) attributed the reversal curse to asymmetries induced by the training dynamics of transformer layers, showing that specific loss functions condition weight updates from one token to another does not necessarily lead the other way round. To mitigate this issue, Golovneva et al. (2024) proposed a reversed training scheme, where models were explicitly fine-tuned with reversed training samples. These works primarily focus on reasoning in posterior setting, where the model is asked to infer "B is entailed by A" given the premise "A entails B" is learned. In contrast, our work shifts attention to priori perspective, demonstrating that equivalence asymmetry arises due to inherent biases in the pre-training data itself specifically, from frequency imbalances in subject-object pairs within the model's prior knowledge.
Reversal Curse
Probing with Knowledge Bases Utilising a knowledge base (KB) to probe LLMs has gained significant interest in recent years, as the structured, high-quality knowledge con- tained a KB provides an effective means to verify a model's understanding and mitigate hallucinations. Petroni et al. (2019) first introduced the concept of "LMs-as-KBs" by crafting several probing datasets from different knowledge graphs (KGs) to examine whether an LLM stores relational knowledge. This approach was later extended to more fine fine-grained KBs, such as temporal KGs (Dhingra et al., 2022) and ontologies (He et al., 2023). In contrast, Zheng et al. (2023) focused on more efficient sampling strategies and alternative metrics to
2

---

Published as a conference paper at COLM 2025
assess knowledge alignment in LLMs. While these studies primarily quantify how much relational knowledge is stored, our work shifts the focus to probing the asymmetry in fact recognition, demonstrating that LLMs may store equivalent facts in an imbalanced manner.
Beyond Long-tail Knowledge Many studies on long-tail knowledge in LLMs focus the model's ability to recognise and retrieve facts about infrequent entities (Kandpal et al., 2023; Sun al., 2024; Li et al., 2024). However, our work extends beyond the traditional long-tail problem by examining cases where subject and object entities exhibit significant discrepancies in pre-training frequency. This difference, rather than overall rarity, leads to systematic inconsistencies in fact recognition. Moreover, while prior works have leveraged Wikidata and other structured knowledge bases to estimate entity frequencies (Wei al., 2023; Chen et al., 2023; Xin et al., 2024), such sources provide only partial view of the pre-training corpus, typically accounting for less than 10% of the total training data (Soldaini et al., 2024). In contrast, we derive more accurate frequency estimates directly from the pre-training data itself, ensuring a more representative analysis of the biases in pre-training affecting LLMs (see Appendix D for further discussion).
3 Problem Formulation
We define two facts fi and f2 as equivalent if they express the same underlying statement. Since statements can appear diverse linguistic forms, we focus on relational facts in KGs to control for linguistic variability and isolate the influence of entity frequencies predictions. Below, we formally define equivalent facts and asymmetry in equivalent facts in the context of KGs, followed by our hypothesis, which empirically verify through probing experiments.
model
Definition 3.1 (Equivalent Facts). In a KG, a relational fact is represented as triple (s, r, o), where is the subject, is the relation, and o is the object. If r is invertible relation such that r(s,0) = -, (o,ss, then (s, o) logically equivalent to (o, r-1, s).
Definition 3.2 (Asymmetry in Equivalent Facts). This refers to the phenomenon in LLMs where P(a (s, r, o)) + P(a (o, r1 s)), even though the two triples are logically equiva- lent. Here, ( -) indicates the model's predictive distribution (Kuhn et al., 2023) over the correctness of the fact.
In this context, the correctness label may be "correct", "incorrect", or "unknown", For our purposes, we focus solely on the "correct" case, operating under the assumption that model judges fact as correct, it recognises the fact; otherwise, it does not.
Our hypothesis is that the observed asymmetry originates from significant differences in the frequencies of the subject and object in the model's pre-training corpus. Specifically, (s) >> count(o), the model is more likely to predict the fact (s, r, o) as correct, i.e.,
P(a = "correct" (s, r, o)) > P(a = "correct" (o, r s)), and conversely, if count(o) > cs) P(a = "correct" (s, r, o)) < P(a = "correct" (o, r-1, 5)), where count(-) denotes the frequency of an entity in the pre-training data.
We primarily focus on symmetric relations for which any differences in the model's responses can be more directly attributed to entity frequency effects. By restricting our analysis these relations, we minimise confounding factors arising from non-symmetric relations (e.g., employedBy vs. employs), which can introduce additional biases due to distinct verbalisations of and -
is r itself (e.g., sibling), ensuring that
4 Methodology
4.1 Indexing Pre-training Corpus
Our objective is to uncover how pre-training biases in LLMs might lead to asymmetry in recognising equivalent facts. Achieving this requires fully open-source models (i.e., open
3

---

Published as a conference paper at COLM 2025
Relation
# Triples
Question Template
Statement Template
twin Town (P190)
39,191
Is twinned with o?
is twinned with o.
spouse (P26)
40,971
Is married to o?
s is married to o.
sibling (P3373)
54,960
Does s have a sibling named o?
s has sibling named o. borders with
bordersWith (P47)
377,967
Does border with o?
Table 1:
Number of triples and natural language templates for symmetric relations extracted from Wikidata5M, where and placeholders for the subject and object, respectively.
weights, training approaches, and datasets), which are rarely available. Among the few meeting these criteria are the OLMo series developed by Ai2 (Groeneveld et al., 2024; OLMo Team et al., 2024). They are pre-trained on Dolma (Soldaini et al., 2024), 11TB open-access corpus, which we index to estimate entity frequencies.
straightforward approach would be apply a Named Entity Recognition (NER) model to Dolma, but our preliminary trials showed that this is either imprecise (e.g,, using a BERT-like NER model) or prohibitively cineccinnuiing (e.g., leveraging a modern decoder-only LLM). Instead, we focus on Wikidata5M (Wang et al., 2021), a relatively high-quality subset of Wikidata (Vrandečié Krötzsch, 2014), and perform string matching for each of its entities. Because each entity in Wikidata5M can have multiple aliases, we simply search the corpus for all possible names of each entity, then sum their occurrences.
However, naive methods such as repeated grep commands on large text files become intractable at Dolma's 'ss scale. Using data structures like Bloom filters (Marone & Van Durme, 2024) allow fast membership testing but do not offer precise frequency counts, whereas suffix arrays (Nasr et al., 2023) often impose a high memory footprint. To address these limitations, we adopt the FM-index (Ferragina & Manzini, 2000; 2005), compressed data structure built on the -Wheeerr Transform (BWT) (Burrows, 1994) that facilitates efficient full-text searches over massive corpora. An FM-index typically consists of: (i) the BWT of the text, which clusters similar substrings to aid searching; (ii) rank and select structures, enabling rapid pattern matching; and (iii) occurrence tables, providing precise frequency counts and location information. By applying this indexing scheme, we compress Dolma into 4TB of indexed files and thus achieve reasonably fast lookups for entity frequencies despite the corpus's considerable size.
We also performed a comparative analysis by querying 100 entities using 64 CPUs with both grep and the FM-index on Dolma. While grep took 20 hours due to full-text scans, the FM-index completed the task just 20 minutes, demonstrating approximately 60x speedup. Additional theoretical analysis is provided in Appendix C.
4.2 Extracting Relational Facts
We extract relational facts, i.e., triples of the form (s, r, o), from Wikidata5M and group them by relation r.3 Since our focus is on symmetric relations, we first identify relations that satisfy the symmetric constraint using the Wikidata Query Service (see Appendix B for the SPARQL query). We then filter these relations by retaining only those with more than 10K triples in Wikidata5M, resulting in six candidate relations.
However, not all of these relations contain sufficient number of triples where high- frequency subjects are paired with low-frequency objects (or vice versa for the inverse relation). After further filtering, we select four symmetric relations, each with at least around 1K triples in the extreme high-to-low and low-to-high divisions, i.e., high-frequency entity count 2 100K and fre-frqyency entity count < 1K. These frequency thresholds are empirical, chosen because it is infeasible to know the complete frequency distribution of all entities in the pre-training corpus. Our analysis shows that among English named
2https: htt.//uggggg..ce..n////// wttaac.......aaaeeeeeeeeeeeeeeee co/datasets/al aelaaaialllaaaaaaeeeaaaaaaaaalel /olmm, version 1.7 was used in our experiments. The file is 4.5IB in its compressed (gzip) form and expands to 11TB when uncompressed.
We focus on entities that have at least one English name; see preprocessing details in Appendix A.
4

---

Published as a conference paper at COLM 2025
Question Prompt
Statement Prompt
+Bsystem]>
<lsystem|>
Please evaluate the statement or claim contained in the question. Respond with only word-elther 'Yes" the
Please evaluate the statement or claim. Respond with only word-either 'Truc' If the claim correct or 'False' it is incorrect. Do not include any additional text or commentary. <luser|>
claim is correct or "No' if it is incorrect. Do not include any additional commentary.
<luseri>
Diego Maradona has sibling named Raul Maradona.
Does Diego Maradona have sibling named Raul Maradona?
Figure 2: A concrete input example for the triple "DiegoMaradona,sibingg RaulMaradona),
shown as a question (left) and as a statement (right). In each case, the system instruction respectively.
restricts the model's response to single word: ""No"""No" or
entities in Wikidata5M, approximately two-thirds have frequencies lower than 1K, while only about 5% exceed 100K pattern that is consistent with the long-tail distribution eupicallyct observed in real-world datasets. Table 1 presents the total number of triples for each selected relation, along with natural language templates for verbalising them (see next section for our probing set-up). The distribution of triples across specific frequency divisions will be reported alongside the results in Section 5.3.
typically
4.3 Probing Asymmetry in Equivalent Facts
Following Definition 3.2, we investigate potential asymmetry in an LLM's predictive dis- tribution: (( (s, r, o)) + P(a (o, s)), where we designate (s, r, o) as the forward triple and (o, r-1, s) the backward triple, both of which are logically equivalent. Since the model processes forward and backward triples through distinct logits due to its autoregres- sive nature, comparing these probabilities directly is not straightforward. Instead, we reveal potential asymmetry by comparing how many forward triples versus backward triples are recognised as correct under the same relation
To assess how an LLM judges the correctness of a relational fact, we frame it as a classification task in which the model needs produce a single-word answer a given the fact's natural language expression and a task instruction. We employ two prompt templates for each fact: question format and a statement format (see Table 1). In the question prompt, the model is asked to respond "Yes" or "No", whereas in the statement prompt, it should answer "True" or "False" 4 A concrete example is provided in Figure 2.
We derive verbalisations for each relation from the corresponding Wikidata descriptions. To better capture the model's potential familiarity with different representations of the same fact, we apply an inference-time scaling approach (Snell et al., 2024), introducing variations of entity names for both subject and object. Specifically, synonyms from Wikidata5M for each entity, resulting in a maximum of 36 prompt variations per fact. Since real-world texts often refer to entities by multiple names, this procedure provides a more robust measure of the model's knowledge. We consider a fact successfully recognised if the model produces the correct one-word label ("Yes" for a question or "True" for a statement) in any of these variations. We apply the same evaluation criterion to both the forward triple (s, 0) and its backward equivalent (o, r-1, s), allowing us to compare recognition accuracies, compute statistical significance, and investigate how entity frequencies affect asymmetry.
randomly select up to six
McNemar's Test To assess the statistical significance of differences between forward and backward triple recognition, we employ McNemar's test (McNemar, 1947), which is designed
for comparing paired data. The test statistic is given by x2 (Nre N
Ny. +Nr where N7F is the number of paired triples recognised in the forward case but not in the backward case, and NFT the opposite. Under the null hypothesis Ho, we have NYF = NET, indicating no
All instruction prompts were generated by GPT-40.
5

---

Published as a conference paper at COLM 2025
asymmetry. The alternative hypothesis Ha posits Nrr + NrT, indicating asymmetry. The p-value is computed as F(x²;1), where F(x; )) is the cumulative distribution function (CDF) of the chi-squared distribution with k degree of freedom. A low p-value (often < 0.05) provides strong evidence reject Ho, showing that the difference in recognition rates is statistically significant and biased in favour of one direction.
5
Experiments
5.1 Models and Implementations
We focus primarily on the OLMo model series due their direct relevance to the pre-training dataset Dolma. In our experiments, we mainly use OLMo2-32B", the most capable variant at the time of our experiments, and also consider the less capable OLMo2-13B* We use the instruction-tuned variants for their ability to follow directions and produce single-word answers, as described in Section 4.3. In addition to the OLMo series, we evaluate Llama3. 1-88? (Dubey et al., 2024) and Qwen2. 5-7B 0wen2.5-7B" (Yang et al., 2024), again using their instruction-tuned versions. Although the extent of overlap between their pre-training data and Dolma remains unknown, we explore whether the entity frequency information from Dolma can be reliable for estimating their pre-training data distributions. For more accurate and viable probing of the LLM's predictive distribution, we report results with temperature = 0.0 (greedy decoding). Our probing pipeline was implemented based on the v1lm² infrastructure for fast inference (Kwon et al., 2023), and all experiments were conducted on H100 GPUs.
5.2 Evaluation Settings
For each relation, we examine three settings: High-to-Low (a high-frequency subject paired with low-frequency object), Low-to-High (a low-frequency subject paired with high- frequency object), and High-to-High (both subject and object are high- (igh-frequency.... 10 In every setting, the forward triples correspond to the original triples from Wikidata5M. Although the relation is symmetric, only one directional instance is typically recorded, meaning that the High-to-Low and Now-to-High settings do not overlap. For the High-to-Low and Low-to-High settings, fix the high-frequency threshold at 100K and consider three low- frequency ranges: 0-1K, 1K-10K, and 10K-100K. In the High-to-High setting, both the subject and the object exceed the 100K threshold. As described in Section 4.3, evaluate two prompt templates (question and statement) to examine the effect of prompt phrasing on fact recognition.
5.3 Results
Results for OLMo Models We present full results for OLMo2-328 in Table 2. Full results for OLMo2-1 3B can be found in Appendix F. Observations for the larger OLMo2-328 32B model are generally consistent with those of OLMo2-13B. For each frequency range, we report the total number of triples, the forward and backward triple recognition accuracies, an arrow symbol indicating the preferred direction (4 for forward and & for backward), and the statistical significance, denoted as: p<0.000 (***), p<0.01 (**), p0005 (*), or not significant (NS).
Overall, the model demonstrates a clear directional preference: favouring forward triples in the High-to-Lov setting and backward triples in the Low-to-High setting. In the most extreme low-frequency range (0-1K), nearly all differences reach top statistical significance (***), with the exception of bordersWi th. Notably, the spouse relation exhibits the largest
Instruct
9https://docs. vllm.
10We omit the Low-to-Loo setting because the sample size and/or o the recognition accuracies in this setting are insufficient to provide statistical meaning.

---

Published as a conference paper at COLM 2025
Question Template
Statement Template
Relation
Low Freq.
Total
Forward
Backward
Diff.
Stat Sig.
Forward
Backward
Diff.
Stat Sig.
High
0-1K
894
0.176
0.110
0.372
0.276
twinned Town
1K-10K
1667
0.219
0.113
0.430
0.347
A
10K-100K
3383
0.238
0.180
0.487
0.469
NS
0-1K
600
0.647
0.383
490
spouse
1K-10K
604
0.734
0.548
10K-100K
0.752
0.727
0.638
0-1K
1707
0.786
0.675
0.767
0.627
sibling
1K-10K
887
0.844
0.796
0.842
0.759
10K-100K
744
0.843
0.836
NS
0.840
0.817
NS
0-IK
12718
NS
0.141
0.135
SN
bordersWith
1K-10K
6..
0.394
0,382
SN
10K-100K
0.480
0.476
Low
High
0-1K
934
0.095
0.171
0.272
0.364
twinned
1K-10K
1674
0.115
0.364
0.444
10K-100K
3465
0.179
0.229
0.463
0,483
0-1K
1064
0,421
0.664
0.374
0.605
spouse
IK-10K
1147
0.581
0,739
0.539
0.727
10K-100K
864
0.652
0.727
0.633
0.725
0-1K
1711
0.677
0.781
444
0.631
0.766
sibling
1K-10K
0.768
0.839
644
0.734
0.844
10K-100K
NS
0.814
0.832
0-1K
13005
0.147
0,148
NS
0.140
0.143
NS
bordersWith
1K-10K
6152
0.389
0.411
0.377
0.386
SN
10K-100K
4418
0.488
0.507
0.474
0.479
NS
High
High
twinnedToww
2 100K
11103
0.231
0.232
NS
0.450
0.451
NS
spouse
2 100K
700
0.666
0.673
NS
0.651
0.660
NS
sibling
100K
754
0.780
0.779
NS
0.776
0.782
NS
bordersWith
> 100K
6254
0.674
0.676
NS
0.635
0.631
NS
Table 2: Results of OLMo2-328 comparing the statistical differences in recognising forward versus backward relational facts using two template types under High-to-Low, Low-to-High, and High-to-Higg settings.
forward-backward accuracy differences, ranging from 0.231 to 0.264 across both templates and settings. While this asymmetry persists in higher low-frequency bands (1K-10K and 10K-100K), it tends to diminish entity frequency increases. For instance, in the High- to-Low setting with the question template, the observed differences for sibling are 0.111, 0.048, and 0.007 for the 0-1K, 1K-10K, and 10K-100K ranges, respectively. Non-significant results (NS) typically appear either in the least extreme frequency range (10K-100K) or for relations that are harder to model, such bordersWith.
We also observe that recognition accuracy generally improves as the fow-frequency range broadens, aligning with general observations on long-tail knowledge. Among the examined relations, twinnedTown and borderswite th consistently yield lower accuracies, likely reflecting the model's greater difficulty with geography-relatee facts (Bhandari et al., 2023).
Finally, in the High-to-High setting where both subject and object entities exceed 100K in frequency, results are consistently non-significant (NS), indicating no clear directional asymmetry when both entities are frequent during pretraining.
Accuracy Ratios across Models Figures and 4 compare the forward bawardd/backward backward recogni- tion accuracy ratios for the spouse and twinnedTown relations across four models: ALMo2-32B,
7

---

Published as a conference paper at COLM 2025
Question Template
forward/Backowa Backkwadd Ratio
1.5
0-1K
1K-10K
10k 100K
100K
Statement Template
accwward Ratio
Forward/s wwward/Backwarre
0.5
1K-10K
100K
110ok
0L (H-41
Owen2
Figure 3: Results of the spouse relation comparing the accuracy ratios across models using two template types under High-to-Low, now-to-High, and High-to-High settings (best viewed in colour).
recognition
OLMo2-1388 1B, Llama3. 1-8B, and Qwen2. 5- These two relations are selected because all mod- els achieve relatively high recognition accuracies, either forward or backward, yielding more stable and meaningful ratios. The red horizontal line at 1.0 denotes parity between forward and backward recognition. For each low-frequency range, both the High-to-Low ratio (plain bar) and the Low-to-High ratio (hatched bar) are presented in the same colour, along with corresponding statistical significance markers above each bar. In the 2 100K frequency range (i.e., High-to-High), the two bars coincide and thus are identical.
Across both relations, we observe a consistent trend: the High-to-Low setting yields forward/back backward ratio greater than while the Low-to-High setting yields a ratio less than 1. For example, for the spouse relation in the 0-1K low-frequency range, OLMo2-13B with the question template achieves High-to-Low ratio of approximately 2.3 and Low- to-High ratio of about 0.4. Notably, even though it is unknown whether Llama3. 1 1-8B and Qwen2.5-78 were pre-trained on Dolma, they exhibit a similar trend with the OLMo models. For instance, both models, using the statement template, attain High-to-Low ratio above 1.2 and Low-to-High ratio around 0.8 for the sibling relation in the 0-1K range. the less extreme low- now-frequeccy range (10K-100K), the forward beewwarddbcckkee backward ratios become much less prominent, with almost no difference observed in the High-to-Hip High setting.
Full results for Llama3. 1-8B and Qwen2.5-2 are available in Appendix F.
6 Discussion
Rationale Behind the Asymmetry We conjecture that the observed asymmetry arises from inherent biases in the pre-training data. In natural language texts, high-frequency entities tend to appear more often as subjects rather than as objects, and subjects typically precede objects in declarative sentences. Consequently, given the autoregressive nature of LLMs, a fact expressed as (s, r, o) with high-frequency entity as and a low-frequency entity as o is more likely to be recognised than its inverted form (o, 11, s). This tendency the training data thus leads to the asymmetry defined in Definition 3.2.
8

---

Published as a conference paper at COLM 2025
Question Template
wwward/Backwwan Ratio
1.0
85
0.8
0.0
0-1K
1K-10K
10k 100K
200K
Statement Template
wwwaadieackkwara ackwward Ratio
1.0
R5
Forwardis
0.6
3K-10K
10k 100K
100K
OLMu2-528 (4-4)
0L (H-4.
1:08
Owen2 (H-4L
OLM62-138
Qwen2.5-78
Figure 4: Results of the sibling relation comparing the newaad/bbacwwer recognition accuracy ratios across models using two template types under High-to-Loww, now-to-High, and High-to- High settings (best viewed in colour).
Classification Rather than Entity Generation common probing setup for similar struc- tural failures in LLMs involves generating free-form answers to a given question (Berglund et al., 2023; Golovneva et al., 2024). In our setting, this would correspond to generating the object given the subject and relation, or generating the subject given the object and the inverse relation. However, this approach essentially examines P(o (s, r, ?)) and P(s (o, (r,,?) which are not expected to be the same unless r is one-to-one. To ensure rigorous probing of the asymmetry in recognising equivalent facts, we instead formu- late the problem as a classification task that aligns with evaluating P(a (s, r, 0)) and P(a (o, ,,, s)), where (s, r, 0) and (o, r-1, s) are logically equivalent.
Detecting Pre-training Data Most state-of-the-ate LLMs do not publicly disclose detailed information about their pre-training datasets. This lack of transparency has raised ethical and legal concerns, driving efforts to infer the composition of pre-training corpora. Various approaches have been proposed, including prompting models to generate data-specific examples (Sainz et al., 2023), employing statistical methods to detect dataset contamination (Golchin & Surdeanu, 2023; Oren et al., 2023), and using membership inference attacks to determine whether specific data points were present during training (Carlini et al., 2021; Shokri et al., 2017). However, these methods primarily rely on model behaviour, such as output probabilities and loss patterns, which do not necessarily provide direct insight into the structure and distribution of the pre-training data itself. In contrast, our work examines inherent characteristics of pre-training corpora that shape an LLM's factual consistency. Through this, we gain clues about the distribution of entities in closed-source pre-training data. Our results on Llama3. 1-8B and Qwen2. 5-7B illustrate that these models also exhibit the frequency-bes -ased asymmetry, suggesting possible distribution, or at least the commonality and rarity, of entity mentions in their undisclosed training data.
Conclusion
In this paper, we investigate the asymmetry in how LLMs recognise logically equivalent facts. Our results indicate that LLMS readily identify relational facts with a high-frequency

---

Published as a conference paper at COLM 2025
subject and low-frequency object but struggle with the inverse form. By leveraging the fully open-source OLMo models and their Dolma pre-training data, we accurately estimate entity frequencies and demonstrate a strong correlation between these frequency discrepancies and model performance. Extending our analysis to models with proprietary pre-training data from the Llama and Qwen families further confirms that the same asymmetry emerges. These findings contribute to understanding how pre-training data characteristics shape model behaviour and motivate future research into mitigating such biases.
Acknowledgments
This work was supported by Samsung Research UK (SRUK), EPSRC projects UK FIRES (EP/S019111/I) and ConCur (EP/V05086) / 11, the DAAD programme Konrad Zuse Schools of Excellence in Artificial Intelligence (relAT), the Federal Ministry of Education and Re- search, the ERC grant AVeriTeC (GA 865958), EU Projects Graph Massivizer (GA 101093202), enRichMyData (GA 101070284) and SMARTY (GA 1011400877 The authors thank the In- ternational Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Yuqicheng Zhu.
Ethics Statement
Our work probes the asymmetrical behaviour of large language models (LLMs) when presented with logically equivalent facts, aiming to shed light potential biases arising from entity frequency in training data. To conduct these experiments, we rely on publicly available, open-licensed data (i.e., Dolma and Wikidata5M) that do not involve collecting or storing personal, private, or sensitive information beyond what is already publicly disclosed. Any real-person examples used (e-g., public figures) appear only to illustrate model behaviour, and do not contain sensitive personal data.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, llge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint Xii::2303008774, 2023.
Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, al. Gemini: a family of highly capable multimodal models. arXiv preprint ar Xrio:2312.118055 2023.
Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. The reversal curse: Llms trained on" a is b" fail to learn" b is a". Xiv preprint ar arXi:2209.112288 2023.
Prabin Bhandari, Antonios Anastasopoulos, and Dieter Pfoser. Are large language models geospatially knowledgeable? In Proceedings of the 31st ACM International Conference on Advances in Geographic Information Systems, pp. 1-4, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901,
2020. 2020.
Michael Burrows. block-sorting lossless data compression algorithm. SRS Research Report, 124, 1994.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Kather- ine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In 30th USENIX security symposium (USENIX Security 21), pp. 2633-2650, 2021.
10

---

Published as a conference paper at COLM 2025
Lihu Chen, Simon Razniewski, and Gerhard Weikum. Knowledge base completion for long-tail entities. In Proceedings of the First Workshop on Matching From Unstructured and Structured Data (MATCHING 2023), pp. 99-108, 2023.
Bhuwan Dhingra, Jeremy R Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William Cohen. Time-aware language models as temporal knowledge bases. Transactions of the Association for Computational Linguistics, 10:257-273, 2022.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama herd of models. arviv preprint ar Xiv:2407. 2024.
Paolo Ferragina and Giovanni Manzini. Opportunistic data structures with applications. In Proceedings 41st amnull symposium on foundations of computer science, pp. 390-398. IEEE, 2000.
Paolo Ferragina and Giovanni Manzini. Indexing compressed text. Journal of the ACM (JACM), 5554)5552- 2005.
Shahriar Golchin and Mihai Surdeanu. Data contamination quiz: A tool detect and estimate contamination in large language models. arXiv preprint ar trXiv:2311.06333 2023.
Olga Golovneva, Zeyuan Allen-Zhu, Jason Weston, and Sainbayar Sukhbaatar. Reverse training to nurse the reversal curse. arviv preprint arXiv:2403. 2024.
Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, et al. Ólmo: Acceler- ating the science of language models. ar preprint ar XrX:i:2402.008.0 00838, 2024.
Yuan He, Jiaoyan Chen, Ernesto Jimenez-Ruiz Hang Dong, and Ian Horrocks. Language model analysis for ontology subsumption inference. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 3439-3453, 2023.
Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems, 43(2):1-55, 2025.
Yue Huang, Lichao Sun, Haoran Wang, Siyuan Wu, Qihui Zhang, Yuan Li, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, et al. Position: Trustllm: Trustworthiness in large language models. In International Conference on Machine Learning, pp. 2266602000 PMLR, 2024.
Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. Large language models struggle to learn long-tail knowledge. International Conference on Machine Learning, pp. 15696-15707. PMLR, 2023.
Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic in- variances for uncertainty estimation in natural language generation. arXio preprint arXin:2302.09664, 2023.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and lon Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium Operating Systems Principles, pp. 611-626, 2023.
Dongyang Li, Junbing Yan, Taolin Zhang, Chengyu Wang, Xiaofeng He, Longtao Huang, Jun Huang, et al. On the role of long-tail knowledge in retrieval augmented large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 120-126, 2024.
11

---

Published as a conference paper at COLM 2025
Fangru Lin, Shaoguang Mao, Emanuele La Malfa, Valentin Hofmann, Adrian de Wynter, Jing Yao, Si-Qing Chen, Michael Wooldridge, and Furu Wei. One language, many gaps: Evaluating dialect faimnsss and robustness of large language models in reasoning tasks. Xiv preprint ar Xix::24400 11005, 2024.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computa- fional Linguistics (Volume 1: Long Papers), pp- 3214-3252, 2022.
Yang Liu, Yuanshun Yao, Jean-Francois Ton, Xiaoying Zhang, Ruocheng Guo, Hao Cheng, Yegor Klochkov, Muhammad Faaiz Taufiq, and Hang Li. Trustworthy llms: sur- vey and guideline for evaluating large language models' alignment. arXiv preprint 0rXiv:2308.05374, 2023.
Marc Marone and Benjamin Van Durme. Data portraits: Recording foundation model training data. Advances in Neural Information Processing Systems, 36, 2024.
Quinn McNemar. Note on the sampling error of the difference between correlated propor- tions or percentages. Psychometrika, 12(2):153-157, 1947.
Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A Feder Cooper, Daphne Ippolito, Christopher A Choquette-Choo Eric Wallace, Florian Tramèr, and Katherine Lee. Scalable extraction of training data from (production) language models. arXiv preprint arXie:2311.17055, 2023.
OLMo Team, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, et al. 2 olmo 2 furious. arXiv preprint XrXiv:2501. 1005666000666...0 2024.
Yonatan Oren, Nicole Meister, Niladri S Chatterji, Faisal Ladhak, and Tatsunori Hashimoto. Proving test set contamination in black-box language models. In The Twelfth International Conference on Learning Representations, 2023.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 1463-2473 2019.
Vipula Rawte, Amit Sheth, and Amitava Das. A survey of hallucination in large foundation models. Xiv preprint ar Xiv:2309.05922, 2023.
Oscar Sainz, Jon Ander Campos, Iker Carcía-Ferrent Julen Etxaniz, Oier Lopez de La- calle, and Eneko Agirre. NIp evaluation in trouble: On the need to measure Ilm data contamination for each benchmark. arviv preprint ar Xio:23100.80118, 2023.
Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models. ar Xiv preprint ar ar1::2310.11678 2023.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEÉE symposium on security and privacy (SP), pp. 3-18. IEEE, 2017.
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute opti- mally can be more effective than scaling model parameters. arviv preprint 4rXiv:2408.03314 2024.
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, et al. Dolma: an open corpus of three trillion tokens for language model pretraining research. In Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume Long Papers), PP- 15725-15788 2024.
12

---

Published as a conference paper at COLM 2025
Kai Sun, Yifan Xu, Hanwen Zha, Yue Liu, and Xin Luna Dong. Head-to-tail: How knowl- edgeable are large language models (llms)? aka will Ilms replace knowledge graphs? In Proceedings of the 2024 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 311-325, 2024.
Denny Vrandečié and Markus Krötzsch. Wikidata: free collaborative knowledgebase. Commuriictioos of the ACM, 57(10)778-85, 2014.
Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhengyan Zhang, Zhiyuan Liu, Juanzi Li, and Jian Tang. Kepler: A unified model for knowledge embedding and pre-trained language representation. Transactions of the Association for Computational Linguistics, 9:176-194, 2021.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
Yanbin Wei, Qiushi Huang, Yu Zhang, and James Kwok. Kicgpt: Large language model with knowledge in context for knowledge graph completion. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 8667-8683 2023.
Amy Xin, Yunjia Qi, Zijun Yao, Fangwei Zhu, Kaisheng Zeng, Xu Bin, Lei Hou, and Juanzi Li. Llmael: Large language models are good context augmenters for entity linking. Xiviv preprint ar Xiv:2407 04020, 2024.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. technical report. arXiv preprint arXiv:2412.15115, 2024.
Shangshang Zheng, He Bai, Yizhe Zhang, Yi Su, Xiaochuan Niu, and Navdeep Jaitly. Kglens: Towards efficient and effective knowledge probing of large language models with knowl- edge graphs. ar Xiv preprint ar XrX::2312.115533 2023.
Hanlin Zhu, Baihe Huang, Shaolun Zhang, Michael Jordan, Jiantao Jiao, Yuandong Tian, and Stuart Russell. Towards a theoretical understanding of the' reversal curse' via training dynamics. Advances in Neural Information Processing Systems, 37:90473-90513, 2024.
13

---

Published as a conference paper at COLM 2025
A Preprocessing of Wikidata5M Entities
Due to the presence of potentially ill-formed entity names Wikidata5M, we apply the following preprocessing steps: (1) remove brackets, underscores, and extra spaces; (ii) retain only names composed of ASCII characters, digits, dashes, periods, commas, quotation marks, and spaces components typically found in English named entities; and (fii) keep only those entities for which at least one valid name remains after the previous steps.
B SPARQL Query for Symmetric Relations
SELECT ?property ?propertyLabel
WHERE {
?property P:32302 ?statement
?statement ps: P2302 wd: wd:021510862 Q21518862
"symmetric constraint"
SERVICE wikibase: label {
bd: serviceaaraa wikibase: language
en
ORDER 2propertyLabel
Listing 1: SPARQL query that retrieves all Wikidata relations (properties) that have the symmetric constraint.
C Efficiency Analysis for Search Algorithms
In our work, we utilise the FM-index for fast entity search in large pre-training corpora, such as Dolma (11TB), owing to its significant advantages in both time and space complexity over traditional grep-based based approaches.
Time Complexity While grep, whether employing the Boyer-Moore or Aho-Corasick algorithms, typically requires scanning the entire corpus for each query, resulting in search time that scales with the corpus size (O(N)), where N represents the total number of characters in the corpus. In contrast, the FM-index performs exact pattern matching in time proportional only to the pattern length (OM)), where M is the length of the search pattern. When dealing with extremely large datasets like Dolma, it becomes practical to partition the FM-index into K separate files. This partitioning allows for parallel processing and more manageable memory requirements during query operations. However, it introduces additional cost for reporting matches, denoted as (K), where is the number of partitions. In practice, since both M and K are typically much smaller than N, the query performance is effectively independent of the corpus size N. Constructing the FM-index involves an initial preprocessing cost, which can be as high O log N). This preprocessing cost is amortised over numerous queries in a static corpus, leading to substantial overall efficiency gains.
Space Complexity Furthermore, by leveraging the Burrows-Wheeld Transform, the FM- index maintains a compressed representation of the data, thereby significantly reducing the memory footprint compared to grep, which must operate on the full, uncompressed text. Partitioning the FM-index into K files also allows for more efficient memory management by loading only the relevant partitions during query execution. This approach ensures that the memory footprint remains manageable, even when working with massive datasets like Dolma. This combination of rapid query performance and memory efficiency makes the FM-index a superior choice for our application.
D Entity Frequency Analysis
We conduct an additional analysis to highlight the importance of directly estimating entity frequency from pre-training data. As noted in Section 2, some prior studies have identified
14

---

Published as a conference paper at COLM 2025
long-tail entities based their rarity in large KGs such as Wikidata. To show that this proxy can be inaccurate, we consider all entities in Wikidata5M and independently estimate their counts in both the Dolma pre-training dataset (see Section 4.1) and Wikidata (via the SPARQL query in Listing 2). Because the raw counts are on different scales, we normalise them using the transformation log(x + 1). We then compute the Pearson linear correlation and Spearman rank correlation between the two sets of normalised counts. The results yield Pearson correlation of 0.291 and Spearman correlation of p 0.255, indicating weak linear relationship between the frequency distributions.
SELECT (COUNT (DISTINCT ?subject) AS ?subject_count) (COUNT (DISTINCT object) AS ?object_count) WHERE {{ {( ?subject ?p wd: (entity_iid)
}} UNION {{
wd: (entityyy_ ?p ?object
}}
}}
Listing 2: SPARQL query that retrieves the total count of an entity (appearing as either subject object) Wikidata.
E Effect of Thinking Before Judging
In the main paper, we focus on prompting models directly judge the truth of factual claim (see Figure 2). Here, we explore whether encouraging the model to think step by step (chain-of-thought (CoT)) prompting (Wei al., 2022)) before making judgment affects its behavior. Specifically, we evaluate OLMo2-328 on the spouse relation (Table 3) using modified instruction: instead of "Respond with only one word...", we prompt the model with "Let's first think step by slep. Afler reasoning, give the final answer-eillar 'Yes' if the claim is correct or 'No' if it is incorrect" (replacing 'Yes' with 'True' and 'No' with 'False' in the Statement Template). As shown in Table 3, CoT does not alter the observed equivalence asymmetry and it remains consistent across frequency ranges.
Question Template
Statement Template
Relation
Low Freq.
Total
Forward
Backward
Diff.
Stat Sig.
Forward
Backward
Diff.
Stat Sig.
High Low
D-1K
1005
0.790
0.684
0.851
0.614
spouse
1K-10K
1141
0.851
0.770
0.889
0.761
10K-100K
858
0.829
0.803
NS
0.893
0.830
Low High
D-1K
1064
0.648
0.747
0.613
0.831
spouse
1K-10K
1147
0.762
0.838
0.753
0.881
10K-100K
864
0.799
0.841
0.815
0.884
High
High
spouse
100K
700
0.803
0.793
NS
0.829
0.831
NS
Table 3: Results of OLMo2- 32B comparing statistical differences in recognising forward and backward relational facts for the spouse relation, using the two modified thinking- before- judging template types under High-to-Low, Low-to-High, and High-to-High settings.
Additional Full Results
In the main paper, we report full results for OLMo2-32B in Table Here, we amend complete results for OLMo2-13B 1BB (Table 4), Llama3. 1-8B (Table 5), and Qwen2. 5-7B (Table 6).
15

---

Published as a conference paper at COLM 2025
To see if the observed asymmetry is consistent with even larger models, conduct further experiments with Llama3. 1-70B (Table 7). The results show that the asymmetry persists in the most extreme freq range (0-1K), with one exception for bordersWith under the state- ment template. In less extreme freq ranges, the model either shows the same trend or no statistically significant difference, likely due to its increased capacity.
Question Template
Statement Template
Relation
Low Freq.
Total
Forward
Backward
Diff.
Stat Sig.
Forward
Backward
Diff.
Stat Sig.
High
Low
D-1K
894
0.032
0.015
0.112
0.088
twinmed Town
1K-10K
1667
0.042
0.018
0.154
0.133
10K-100K
3383
0.076
0.047
0.219
0.213
0-1K
1005
0.337
0.146
6.4
0,380
0.256
1K-10K
1141
0.472
0.314
0.616
0,492
10K- 100K
858
0.565
0.488
0.681
0.634
0-1K
1707
0.408
0.374
0.583
0.535
sibling
1K-10K
0.626
0.539
0.745
0.649
10K-100K
744
0.638
0.608
0.712
0.695
NS
0-IK
12718
0.031
0.022
0.059
0.054
bardersWith
IK-10K
6132
0.140
0.129
0.211
0.186
10K-100K
4397
0.296
0.281
0.371
0.357
Low High
U-1K
0.092
twinnedTowe
IK-10K
1674
0.018
0.051
0.136
0.156
10K-100K
3465
0.052
0.078
0.211
0.225
0-1K
1064
0.148
0.339
0.243
0.387
spouse
1K-10K
1147
0,316
0.471
0.476
0.595
10K-100K
864
0.479
0.557
0.625
0.679
0-1K
1711
0,373
0.402
0.538
0.563
sibling
1K-1KK
0.524
0.621
4.4
0.638
0.738
10K-
752
0.616
0.637
NS
0.682
0.694
NS
U-1K
13005
0.023
0.031
0.058
0.060
NS
bordersWith
1K-10K
6152
0.123
0.136
0.181
0.203
10K-100K
4418
0.281
0.290
NS
0.354
0.366
NS
High
High
twinned Town
>> 100K
11103
0.146
0.150
NS
0.345
0.348
NS
spouse
2 100K
0.563
0.563
0.647
0.641
sibling
2 100K
754
0.569
0.564
NS
0.601
0.618
NS
bordersWith
100K
6254
0.554
0.553
NS
0.600
0.596
Table 4: Results of OLMo2- 13B comparing the statistical differences in recognising forward versus backward relational facts using two template types under High-to-Low, Low-to-High, and High-to-High settings.
hlnttpss//hugggiggaac.cccc///// 1 Lammaaammmmmmaaaaamamm 108 Instruct
16

---

Published as a conference paper at COLM 2025
Question Template
Statement Template
Relation
Low Freq.
Total
Forward
Backward
Diff.
Stat Sig.
Forward
Backward
Diff.
Stat Sig.
High
0-1K
894
0.888
0.868
NS
0.714
0.655
twinned Town
1K-10K
1667
0.901
0,878
0.754
0.669
A
10K-100K
3383
0.952
0,938
0.791
0.741
0-1K
1005
0.737
0.554
0.623
0.444
spouse
404
0.654
10K-100K
0.783
0.763
0.723
0-1K
1707
0.884
0.692
0.813
0.667
sibling
1K-10K
887
0.924
0.763
D.868
0.763
10K-100K
744
0.910
0.637
0.835
0.789
12718
404
0.159
0.191
bordersWith
1K-10K
6132
0.646
0.330
0.311
10K-100K
NS
Low
High
D-1K
934
0.874
0.583
NS
0.654
0.713
twinned
1K-10K
0.671
0,754
10K-100K
3465
0.943
0.952
0.744
0.794
0-1K
1064
0.523
0,714
0.429
0.606
spouse
1K-10K
1147
0.712
0,810
0.650
0.765
10K-100K
864
0.786
0,814
NS
0.726
0.751
NS
0-1K
0.687
444
0.654
0.819
sibling
1K-10K
0.926
0.757
0.865
10K-100K
404
0-1K
13005
0.463
0.416
.*e 0.190
0.164
bordersWith
1K-10K
6152
0.655
0.645
NS
0.313
0.330
100K
4418
0.691
0.693
NS
D.398
0.382
High
High
twinnedToww
100K
11103
0.932
0.932
NS
0.697
0.693
NS
spouse
100K
700
0.747
0.750
NS
0.681
0.674
NS
sibling
100K
0.871
0.877
NS
0.772
0.773
NS
bordersWith
> 100K
6254
0.784
0.783
NS
0.516
0.518
NS
Table 5; Results of Llama3. 1-8B comparing the statistical differences in recognising forward versus backward relational facts using two template types under High-to-Low, Low-to-High, and High-to-High settings.
17

---

Published as a conference paper at COLM 2025
Question Template
Statement Template
Relation
Low Freq.
Total
Forward
Backward
Diff.
Stat Sig.
Forward
Backward
Diff.
Stat Sig.
High
0-1K
894
0.393
0,375
NS
0.834
0.856
NS
twinned Town
1K-10K
1667
0.490
0,379
0.916
0.878
10K-100K
3383
0.565
0.469
0.948
0.915
0-1K
0.320
0.221
0.687
0.544
spouse
404
0.810
0.623
10K-100K
649
0.728
0-1K
1707
0.582
0.449
0.783
0.640
sibling
1K-10K
887
0.667
0.506
0.830
0.669
10K-100K
744
0.726
0.624
0.804
0.738
12718
404
0.219
0.281
bordersWith
1K-10K
6132
0.171
NS
0.445
0.466
10K-100K
6..
0.555
Low
High
D-1K
934
0.366
0.387
NS
0.857
0.833
1K-10K
0.381
0.505
404
0.876
0.924
10K-100K
3465
0.468
0.568
464
0.916
0.947
0-1K
1064
0.209
0,308
0.525
0.661
spouse
1K-10K
1147
0.253
0.437
0.626
0.791
10K-100K
864
0,376
0.481
0.728
0.788
0-1K
0.449
0.564
444
0.634
0.788
sibling
1K-10K
0.506
0.664
0.670
0.813
10K-100K
404
0.743
0-1K
13005
0.061
0.047
0.285
0.219
bordersWith
1K-10K
6152
0.166
0.167
NS
0.424
0.441
10KK
4418
0.336
0.299
0.563
0.552
NS
High
High
twinnedToww
100K
11103
0.443
0.445
NS
0.869
0.869
NS
spouse
100K
700
0.501
0.524
NS
0.761
0.773
NS
sibling
100K
0.678
NS
0.744
0.735
NS
bordersWith
> 100K
6254
0.531
0.530
NS
0.688
0.688
NS
Table 6: Results of Qwen2.5-25 comparing the statistical differences in recognising forward versus backward relational facts using two template types under High-to-Low, Low-to-High, and High-to-High settings.
18

---

Published as a conference paper at COLM 2025
Question Template
Statement Template
Relation
Low Freq.
Total
Forward
Backward
Diff.
Stat Sig.
Forward
Backward
Diff.
Stat Sig.
High
0-1K
894
0.968
0.938
0.971
0.932
twinned Town
1K-10K
1667
0.974
0,945
0.972
0.936
10K-100K
3383
0.982
0.947
0.973
0.937
0-1K
1005
0.902
0.834
644
0.569
0.826
spouse
10K-100K
NS
0.850
0-1K
1707
0,941
0.884
0.948
0.902
sibling
1K-10K
887
0.968
0.939
0.964
0.950
NS
10K-100K
744
0.965
0.941
0.958
0.950
NS
12718
404
0.863
0.870
bordersWith
1K-10K
6132
0.927
0,924
10K-100K
Low
High
D-1K
934
0.934
0.967
0.924
0,970
1K-10K
1674
0.928
0.969
10K-100K
3465
0.946
0.979
0.933
0.973
0-1K
1064
0.817
0.867
0.809
0.837
spouse
IK-10K
1147
0.891
0.887
NS
0.987
0.875
NS
10K-100K
864
0.846
0.826
NS
0.852
0.829
NS
0-1K
1711
0.879
444
0.897
0.938
sibling
1K-10K
0.926
0.958
644
0.941
0.961
10K-100K
0.952
0.953
0-1K
13005
0.909
0.919
***
0.872
0.864
bordersWith
1K-10K
6152
0.938
0.950
0.921
0.928
10KK
4418
0.932
0.936
NS
0.913
0.917
NS
High
High
twinnedToww
100K
11103
0.945
0.947
NS
0.923
0.926
NS
spouse
100K
700
0.791
0.806
NS
0.789
0.794
NS
sibling
100K
754
0.939
0.938
NS
0.952
0.955
NS
bordersWith
> 100K
6254
0.928
0.925
NS
0.903
0.904
NS
Table 7: Results of Llama3. 1 1-70B comparing the statistical differences in recognising forward versus backward relational facts using two template types under High-to-Low, Low-to-High, and High-to-Hight settings.
19


Reviews:
[{"review_id": "eyKgGwUIlI", "reviewer": "Reviewer_HfnW", "summary": "This work examines the influence of entity frequency in LLM pretraining corpora on the ability to recognize factual statements. In particular, they examine symmetric subject, relation, object tuples from the Wikidata knowledge base where the statement is true for either ordering of the subject and object entities (e.g., \"A boarders B\", \"A is the sibling of B\"). The authors track effects of each entity's frequency in the Dolma pretraining corpus their ordering in the statement (i.e., whether the more frequent entity is subject or object) on the ability for LMs to recognize whether the statement is true or false. The authors find that LMs are significantly better at assessing the factuality of statements when the more common entity is given as the subject of the statement.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This work provides insight into a well known problem in LLM training where models that learn to recognize factual statements with one entity ordering fail to also learn (i.e., models that learn that \"A is B\" fail to also recognize \"B is A\").\n\nThe work is fairly thorough, testing multiple LMs including the OLMo family of models, which are pretrained on the exact Dolma corpus examined in this work, as well as the Qwen family of models, where pretraining datasets are unknown. The authors also perform their experiments with 2 prompt templates, and find similar results in both settings.", "reasons_to_reject": "One concern is with the scope of the experiments. While the total number of examples is large, this work analyzes only 4 relations in WikiData. While trends are consistent across these 4 relations, this limited scope casts some concerns regarding the generality of the observed trends. Furthermore, the work only examines LMs up to the 32B scale. Experiments with larger LMs (e.g., API-based Claude, GPT, Gemini) may also yield substantially different trends.\n\nAnother potential concern is that the factuality recognition task used in this work is that all tested statements are exclusively true. This lead to results being conflated with LM biases in multiple-choice scenarios. The tested LMs, furthermore, are exclusively Instruciton-tuned models which may have learned refusal / hallucination-prevention behaviors like \"be less confident in facts that are about uncommon entities.", "questions_to_authors": "Is there a significant reason why treating settings like \"high -> low (forward)\" and \"low -> high (backward)\" as distinct settings? The only difference here, as I understand, is how the direction that the relation is stored in WikiData. Is there a common trend in the ordering that Wikidata stores such symmetric relations that would cause us to expect different results in these settings?", "extra_scores": {}}, {"review_id": "Yq4eXOPl6p", "reviewer": "Reviewer_Ubaa", "summary": "The paper investigates the hypothesis that the \"reversal curse\" of Berglund et\nal.---that an LLM that can correctly answer the question \"who was X's\nmother?\" with \"Y\" cannot give the answer the symmetrically entailed answer\n\"X\" to the question  \"Who was Y's child?\"---has its origin in a\nwell-known bias in LLM QA towards higher frequency entities.\n\nThe method is to extract subject-relation-object triples for frequent\nsymmetrical relations from WikiData5M, then to assess the relative\nfrequency of  subject and object in the DoLMa pretraining dataset for\nthe OLMo LLM, and to separate the triples into those with higher\nfrequency subjects than objects (H -> L) versus the reverse (L -> H),\nand the case where all entities are high-frequency (H->H).\nIt is noted that there is a bias in the DoLMa data towards suject\nfrequency exceeding object frequency, so presumably the absolute\nnumbers of triples in the three sets are different, though I couldn't\nfind those numbers reported.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "An interresting property of this selection is that, if LLM QA were\nworking purely by memorizing the training data, then it is highly\nlikely that the (for example) the L->H triples are actually attested\nin the pretraining data, while their inverses are not.   The extent to\nwhich the LLM performs better on the true but unattested inverses than\non the attested originals is a measure of the strength of the frequency\nbias.   As far as I can see, the analysis of which triples are\nactually attested in DoLMa has not been done, but it\ndoes make this a very interesting dataset, since it potentially\nseparates the frequency bias from another well-known factor, the\nattestation bias.   I think more could have been made of this point.\n\nFor each of the three sets, the success-rate of OLMo (and other\nmodels) in answering a question based on the triple is compared with\nthat on its symmetrically-entailed inverse.  As predicted by the\nfrequency bias, H->L questions are more succesfully answered than\ntheir inverses, L->H inverses are more successfully answered than\ntheir originals, and H->H questions are about the same in both\ndirections.\n\nThere is an interesting review of work specifically on the reversal\ncurse and LLM-as-KB,, including Zue et al and and Petroni.", "reasons_to_reject": "The motivation for the dataset construction could have been clearer.  In\nthe end I wasn't sure what the design behind it actually was.\n\nGiven the considerable amount of work out there on the frequency bias\nin LMs, I think the results are not too surprising:  their significance\nlies in the fact that  they are  backed up by actual frequency counts in\nthe actuual pretraining data for the OLMo LM, making for an\ninteresting dataset.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "30dE44587A", "reviewer": "Reviewer_WBVB", "summary": "This paper investigates a subtle but impactful source of factual inconsistency in large language models (LLMs): asymmetry in recognizing logically equivalent relational facts due to subject-object frequency imbalance during pre-training. The authors use fully open-source models (OLMo-2 series) and their associated Dolma corpus, to estimate entity frequencies with an FM-index and probe models’ ability to recognize facts and their inverse forms. The study confirms LLMs are significantly more likely to recognize high-frequency-to-low-frequency triples than their inverse for symmetric and equivalent relations. Different from previous model-only study, the paper offers a novel and careful empirical analysis from pre-training data distribution, and links it to model factual behavior, with implications for understanding hallucination and knowledge asymmetry in LLMs.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "## Empiricism, Data, and Evaluation\n- Uses realistic data from Wikidata5M, and precise frequency estimation from 11TB Dolma Pre-training corpus. It probes a wide range of symmetric relations with large triple sets across various frequency bands. \n\n- The evaluation links data to model performance (OLMo) and analyzes other models LLaMA3.1, Qwen2.5). It does some statistical tests (McNemar’s test). The design controls for confounding variables (relation symmetry, verbalization variance) well.\n\n\n## Technical Depth & Impact \n- The authors provide a principled definition of equivalent fact asymmetry (Definition 3.2). Their methodological rigor in isolating subject/object frequency as the main variable is impressive. Meanwhile, the method takes into account the efficiency which naturally fits for the scale of pre-training data.\n\n- Insightfully connects a well-known symptom (“reversal curse”) to a Pre-training data-level root cause. \nThe framing is not radical, but the idea of “fact asymmetry from data frequency” is relatively novel and opens doors to future directions, e.g. Extension to asymmetric relations; Applications to data attribution, hallucination diagnosis, and LLM alignment auditing.\n\n## Clarity, Honesty, and Trust\nThe paper is clearly written, with thoughtful diagrams (e.g., forward/backward accuracy ratios).  Care is taken to explain limitations, such as not probing non-symmetric relations. Code and data are open-sourced.", "reasons_to_reject": "- The work deliberately avoids probing non-symmetric relations, which are arguably more common and interesting in natural language (natural power-low, e.g., “employed by”). The rationale is valid, but it limits generalizability.  The analysis focuses on binary classification which is suitable for symmetric relations, but not fit for asymmetries.\n\n- There is no direct proposal for mitigation — e.g., how one might use this insight to improve pre-training or fine-tuning processes.", "questions_to_authors": "1. Address generalization to asymmetric relations in future work — perhaps via a more nuanced probing framework.\n\n2. Broaden discussion on how to mitigate asymmetry — e.g., whether rebalancing subject/object frequencies in fine-tuning could help.", "extra_scores": {}}, {"review_id": "KMWbuOqRCC", "reviewer": "Reviewer_H939", "summary": "This paper focuses on a research question of “how do entity frequencies affect the LLM’s knowledge behavior”, and conduct corresponding analyses. To alleviate relation’s side effects (reversal relation expressions), this paper only focuses on asymmetry relations. The authors construct efficient indexing on Dolma with OLMo series models, which increases the research transparency.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper validate hypotheses from a data perspective, which is very straightforward and convincing.\n- The dataset indexing solution is practical for other pre-training data analyses use cases.", "reasons_to_reject": "- Since pre-training data are directly related to the model behavior, it is essential to analyze the data. However, only one aspect (frequency) is mentioned in this paper, while others are omitted, such as knowledge correctness, knowledge conflict, multi-lingual problems, etc.\n- Although this paper provides empirical analyses about the relationships between training data frequency and reversal curse, it is not clear how these findings would help mitigate such an issue.\n- Besides the above aspects, the following research questions may be further addressed:\n    - The effects on different entity types (e.g. person, organization, location) and relation types.\n    - The effects from entity ambiguities. Do entities with more ambiguities (entities with the same names) lead to hallucinations?\n- There are some potential evaluation biases:\n    - Why do the evaluation prompts constrain the response to only one word instead of generating the thinking process?\n    - There is no background knowledge about the two entities (s and o). How do you ensure the response is 100% correct? Maybe there are different answers under different entities with the same name. For example, we cannot answer “Does Jack have a sibling named Peter?” because there may be a real sibling relationship in the world even they are not famous celebrities.\n\nI may misunderstand some settings or miss some details, please feel free to give me some feedback during the discussion period :)", "questions_to_authors": "- How to address the entity linking (or disambiguation) problem? For example, there may be multiple entities with the same name.\n- I’m not sure if the filtering process in line 167-168 is reasonable, since the number of KG triples may not represent the number of appearances in the pre-training data.", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper raises an interesting and well-executed empirical finding: that LLMs show asymmetry in recognizing logically equivalent relational facts depending on entity frequency. But the scope remains narrow—limited to only symmetric relations and frequency as the sole factor. There’s no compelling strategy for mitigation, and the paper stops short of broadening its findings to more realistic, asymmetric scenarios or entity ambiguity issues. While the rebuttal adds helpful clarifications and new results with larger models and CoT prompting, it doesn’t fundamentally address these limitations. Overall, it’s a thoughtful analysis, and while there remains much to be desired, this is a challenging space, and this paper makes a meaningful contribution. 

Decision: Accept