GeoILP: A Synthetic Dataset to Guide Large-Scale Rule Induction
Authors: Si Chen
Richong Zhang
Xu Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: dataset
inductive logic programming
rule induction
neuro-symbolic methods
Primary area: datasets and benchmarks

Abstract: Inductive logic programming (ILP) is a machine learning approach aiming to learn explanatory rules from data.
    While existing ILP systems can successfully solve small-scale tasks, large-scale applications with various language biases are rarely explored.
    Besides, it is crucial for a large majority of current ILP systems to require expert-defined language bias, which hampers the development of ILP towards broader utilizations.
    In this paper, we introduce GeoILP, a large-scale synthetic dataset of diverse ILP tasks involving numerous aspects of language bias.
    These tasks are built from geometry problems, at the level from textbook exercise to regional International Mathematical Olympiad (IMO), with the help of a deduction engine.
    These problems are elaborately selected to cover all challenging language biases, such as recursion, predicate invention, and high arity.
    Experimental results show that no existing method can solve GeoILP tasks.
    In addition, along with classic symbolic-form data, we provide image-form data to boost the development of the joint learning of neural perception and symbolic rule induction.

Published as a conference paper at ICLR 2025
GEOILP: A SYNTHETIC DATASET TO GUIDE LARGE-SCALE RULE INDUCTION
Si Chen !, Richong Zhang 1.2 Xu Zhang
1 SKLCCSE, Beihang University, Beijing, China
2 Zhongguaneun Laboratory, Beijing. China
3 The National Computer Network Emergency Response Technical Team Coordination
Center of China (CNCERT/CC)
chen. si @buaa. edu. en, zhangro@act, buaa. edu. cn, cn, zhangxu@cert.orgg org cn
ABSTRACT
Inductive logic programming (ILP) a machine learning approach aiming to learn explanatory rules from data. While existing ILP systems successfully solve small-scale tasks, large-scale applications with various language biases are rarely explored. Besides, is crucial for large majority of current ILP systems require expert-defined language bias, which hampers the development of ILP to- wards broader utilizations. In this paper, we introduce GeolL., a large-scale syn- thetic dataset of diverse ILP tasks involving numerous aspects of language bias. These tasks are built from geometry problems, at the level from textbook exercise to regional International Mathematical Olympiad (IMO), with the help of a de- duction engine. These problems are elaborately selected cover all challenging language biases, such as recursion, predicate invention, and high arity. Experi- mental results show that no existing method can solve GeolLP tasks. In addition, along with classic symbolic-forn data, provide image-form data to boost the development of the joint learning of neural perception and symbolic rule induc- tion.
INTRODUCTION
Inductive logic programming (ILP), at the intersection of machine learning (ML) and symbolic tificial intelligence, learns hypotheses from background knowledge and examples (Muggleton & De Raedt 1994 Muggleton al., 2012 Cropper et al.) 2020a] Cropper & Dumančić| [022 Zhang et al. (023).. IIP adopts logical formulae to represent knowledge, examples, and hypotheses uni- formly. The most fascinating merit of II.P, differing from other MI. approaches, the ability to learn highly interpretable hypotheses, which reveals potential way toward human-compprhenniill. controllable, and woothh artificial intelligence.
Classic symbolic ILP are based discrete search, suffering from the combinatorially growing search space and thus restricting to small-scale scenes. To alleviate this obstacle, symbolic meth- ods require user-defined language bias to limit searching, which is markedly crucial for efficiency (Cropper & Dumančić 2022). Such hand-crafted work is more or less the same as the feature engi- neering (Khalid al. (214) other MI. tasks, requiring certain expert knowledge and considerably many troublesome trial and error. However, in the modern MI. community, feature engineering is usually superseded by automatic feature extractors, such as various neural networks, which achieve amazing success in large-scale applications (e.g., GPT-4 (Achiam et al. (003)). Consequently, we argue that thoroughly turning hand-crafe crafted determination language bias into automatic language bias discovery is promising direction towards broader applications of ILP.
Modern neural-symbolic ILP relaxes the hypotheses space into continuous space and leverage gradient-based optimization techniques to induce solutions, from which interpretable rules can be extracted. Despite not requiring an elaborated language bias, existing neural-symbolic methods are limited to relatively small hypotheses space, presuming low-arity predicates, function-free clauses,
"Corresoondigg author

---

Published as a conference paper at ICLR 2025
and few rule's body atoms (Glanois et al. 2022). Scaling up to large- scale scenarios is also of the major challenges of this line of work.
However, large-scale ILP datasets are lacking in evaluating enhancement. Existing datasets are small or lack reference hypotheses. Our goal is to construct a large-scale ILP dataset, providing reference hypotheses to guide the resolution of present limitations in ILP, which would lead ILP to an expert-free learning paradigm (like modern deep learning) and exceedingly broader utilization. Furthermore, we aim evaluate ILP systems without much expert priors (like other modern ML paradigms), i.e., training & testing without excessive user-defined bias.
powerful methods and guiding
Therefore, we construct GeolLP large-scale dataset synthesized from plane geometry rules that help generate reference hypotheses involving various language biases. We first adopt a symbolic deduction engine to obtain target examples from the rules and determine the background knowledge and hypotheses by tracing back from the examples. We also consider the noisy and multi-task settings, which are closer to real-world applications and actively studied in other ML tasks.
In summary, GeolLP contains 835 single-tasks and 207 multi-tasks. The predicate arity is up to and the number of body atoms is up to 9. Overall, 85% single-tasks and 50% multi-tasks contain hypothesis with the number of rules ranging from at least 10 to 100, while the rest of the tasks involve at least hundreds to thousands of rules (refer to section 5.3 for details). Besides, the language biases also involve argument symmetry, constraints, different types of recursion, and predicate invention. We conduct experiments applicable methods, showing that GeolLP is completely unreachable.
In addition, GeolLP provides image-form background knowledge, which requires jointly training perception network, transforming the raw sensory input (image) into symbolic knowledge, and an ILP system inducing the hypothesis. The breakthrough for such joint learning, which remains less explored, would be breakthrough for the whole artificial intelligence community (Cropper & Dumančió 2022).
RELATED WORK
2.1 ILP METHODS
Symbolic methods search in the hypotheses space defined by language biases. Among these, notable methods include FOIL (Quinlan 1990). Progol (Muggleton) 1995 Muggleton Bryant
2000). TILED (Blockeel & De Raedi 1997). ALEPH (SRINIVASAN 2001), Metagol (Muggleton
ct al. 2015), ILASP (Law et al. 2018 2020). These methods suffer from combinatorially growing hypotheses, noisy data, and inefficient predicate invention. Popper (Cropper & Morel 202 is modern symbolic ILP system, which is, to the best of our knowledge, the only symbolic system capable of simultancously learning recursive rules, inventing predicates (Cropper & Morel 2021b), handling noise (Hocquette et al. 2024), and scaling better, though still very expensive to do these.
Neural-symbolic methods differentiable methods, make continuous relaxation of the discrete hypotheses space and induce solutions by minimizing loss function via gradient-based optimizer. While the early-stage methods require user-defined language templates task-by-task to restrict hy- potheses (Rocktäschel & Riedel| 2017] [Campero et 2018))) the following works tend to automat- ically deal with more general language biases (Evans & Grefenstette 2018 Si al. 2019 Glanois et al. 2022). As learning interpretable solutions is the outstanding property of ILP, the methods that cannot produce human-readabe rules are out of the scope of this paper (e.g.. Dong et (019
2.2 ILP DATASETS
Real- world datasets Real-world datasets collect background knowledge and examples from real- world observation. The application scenarios cover knowledge base completion (Bordes et al. 2013 Toutanova & Chen 2015 Yang al. 2017 Hudson & Manning 2019), drug design (Inoue al. 2013
Tamaddoni-Nezhad et al. 2006), ecology (Bohan et al. 2017). etc. The main demerit of
Data is available atttpps:///gh www/chonc/aaeeeeece 99/9999/
2

---

Published as a conference paper at ICLR 2025
real-world datasets is lacking reference hypotheses. Consequently, an ILP system failing these datasets would have little idea about where to improve.
Synthetic datasets
Synthetic datasets covering mathematical formal systems (Evans & Grefen- Stette 2018), grammar learning (Muggleton et al., 2014 Law et al.) 2019), games (Cropper et al,, 2020b), program analysis (Sivaraman et al. 2019 Bartha & Cheney 2020), etc. They can provide reference hypotheses to guide resolving the limitations of ILP systems. However, current synthetic datasets are small-scale, whose hypotheses typically contain less than 10 rules, and have already been solved by existing II.. Our work extends this line of work to much larger scenarios.
3 BACKGROUND
We first introduce necessary logic notions and then define inductive logic programming (ILP). Fur- ther terminology is illustrated in the next section as well.
3.1 LOGIC PRELIMINARIES
Horn clause Every formula in first-order language can be transformed into its semantically equiv- alent conjunctive normal form, a conjunction clauses. A clause is a disjunction of literals. A literal is an atom (positive literal) or its negation (negative literal). An atom is called ground if it contains no variable. Hom clause is a widely used subset of clauses that allow at most one positive literal. Horn clause involve facts, which atoms, and rules that can be semantically equivalently represented as (assumed function-fres here)
H(X)= B, (X) (x A...A (x ) is the head atom of the rule, and the atoms B, (X), BX)), .... (X) the body atoms of the rule. A program is set of Horn clauses. We
where X denotes a vector of variables, the atom H(X)
define rule size as the number of atoms in rule and program size as the sum of rule size.
Note that the variables in a clause are implicitly quantified by universal quantifiers that are supposed to be placed at the beginning. The variables appearing only in the body but not the head are called existentially quantified.
Forward chaining Forward chaining can be used to deduce all the true ground facts from given rules and background facts. Formally, given set of ground atoms A and Horn rule set R, the immediate consequence through one-step forward chaining is defined as the set
A a. E EA
k
conR (A) AU a , Or ground
1=1
where ground(R) consists of all the ground rules instantiated from R. Then, we recursively define the consequence through t steps CR,
CR,O(A) A, CR. con (A)
We say the fix point is reached at step T if is the smallest natural number satisfying CR,T(A) CR,T+1 (A)), and (xA) is the set of all the consequences of the forward chaining.
3.2 INDUCTIVE LOGIC PROGRAMMING
We adopt the most popular ILP setting learning from entailment (LFE) (Cropper & Dumančid 2022). The training data is a tuple (B, E+, E-- of background knowledge B. positive examples of the concept E+, and negative examples of the concept - 3 E are sets of ground atoms relevant to the target predicate we want to learn. B is a set of clauses that act as background knowledge (BK), typically a set of ground atoms irrelevant the target predicate (rules can also be in BK). The goal of ILP is to induce a hypothesis H. consisting also of clauses, satisfying the following conditions
Ve E 8+ HUBFC
(completeness)
Ve E E HUBKE
(consistency)
3

---

Published as a conference paper at ICLR 2025
The completeness condition states that the hypothesis and BK entail all positive examples. The consistency condition states that the hypothesis and BK do not entail any negative examples.
For example. given B [Grandfather (Tom, Ma E- =
[Father (John, Mary), Father(Tom, John)), E+ (earaay Tom), Grandfather(John Mary)), an ILP task learner may learn the following hypothesis H (upper italic letters denote variables)
Grandfather(X, V) Father X, Z) Father YZ
With the learned hypothesis, an automated theorem prover can derive all the facts regarding the target predicate Grandfather from the background facts regarding the relation Father.
4 LIMITATIONS OF CURRENT ILP
In this section, we identify the critical limitations of current II. that impede the development of broader applications. Our proposed dataset is intended to cover all these challenges and is thus good testbed for elaborating more sophisticated rule induction systems.
4.1 HAND-CRAFTED LANGUAGE BIAS
Language bias used limit the hypothesis space in symbolic ILP. As calculated by Cropper & Morel 202 aa), the number of possible hypotheses grows combinatorially fast.
Without carefully nutanndetermined language bias, such as the predicates allowed to appear in the rule's head, the predicates allowed to appear in the rule's body, enabling recursion or not, enabling predicate invention or not, the maximum number of clauses allowed in hypothesis, the maximum number of unique variables in a clause, the maximum number of body atoms in clause, the number of allowed existentially quantified variables, the maximum times a predicate can appear in a rule, symbolic ILP tends to be extremely slow, even useless (Cropper & Dumančić 2022). Determining a good language bias is onerous and requires a vast amount of trial and error.
Below, we introduce the most dominant language biases, which notably increase hypothesis space and should thus be completely automatically determined by ILP systems.
Predicate arity Real-world relations may involve several entities. For instance, the triadic relation
Sell(selle, buyer, book) asserts that, in transaction order, seller sells book buyer. However, current neuro-symbolic methods typically support arity lower than two (Evans & Grefenstette 20118] Campero al. 2018 Glanois et al., 2022). Several symbolic methods can support arbitrary arity but exceedingly increase search complexity CCopper & Morel 2021a).
Argument symmetry Argument symmetry may exist for predicates. For example, if John is Mary's 's cousin, then Mary must also be John's cousin. As this example, argument symmetry can be represented as Horn rule, yet complex symmetry may yield too many rules. For instance, the triadic atom asserting whether people queue straight line evaluates to the same truth value permuting all 3 arguments (any 3 people), which yields 3!(3! 1) 30 Horn rules. To the best of our knowledge, there is no specialized way to learn compact representations for argument symmetry.
Predicate constraint Atoms' truth value may be forced to be opposite under some constraints. Asymmetry constraint can be considered Horn goal (clause with only negative literals) <- Pred(X, Y) A PY) The representation transitivity, anti-triangularity, functionality, exclusivity) can be found in Cropper & Hocquette 2023). Current neuro-symbolic methods do not cover this aspect.
other constraints (e.g., irreflexivity, anti-
Recursion There are two types of recursion in Horn programs: recursion and mutual recursion (Bancilhon & Ramakrishnan 1986). Recursion refers to the phenomenon that the same predicate ap- pears simultaneously in a rule's 's head and body. For instance, Even(X) <- Even (Y') A Suuce(Y,,X' is recursive, where Even asserts whether natural number is even and Sucez X) asserts whether X Y + 2. Even is the recursive predicate in this case. Besides, this recursive rule is also called linear because the recursive predicate only appears once in the body. Mutual recursion refers the
4

---

Published as a conference paper at ICLR 2025
phenomenon that two predicates mutually derive from each other. We say predicate Pred, derives another predicate Pred2 there exists such a set of rules (variables omitted)
Pred2 f- Qı Q f Q2 Q2 x..... Q3
Qn t.... A Predj
where Qs denote other predicates and ... denote any other body atoms. Therefore, two mutually recursive predicates can be mutually deduced via forward chaining. For instance,
Even (X +- Odd(Y) A SYco(Y,X) X)
Odd(X) +- Ev) A Suco(Y,
where Suce(X; (x, Y) asserts whether Y = X + 1, show that Even and Odd (asserting whether natural number is odd) are mutually recursive. A rule also called recursive the head predicate is mutually recursive with one of its body predicates, and the rule is called linear if only one mutually recursive predicate appears in the body. Enabling recursion expensive for symbolic IIP, while mutual recursion between any two predicates is not supported by the ofte-of-thhe-eee neuro-symbolic ILP Glanois et al.) 2022).
Predicate invention Predicate invention is crucial part of automatically discovering new concepts, which may lead to breakthroughs in Al development [Russell 2019 chap. 3). Specifically, predicate invention enables predicates that unused in BK & target examples appearing in the hypothesis. For example, learning Even from the BK (O)ro(0), Succ(O, 1), Succ(11, Succ(2,)),.. may require inventing dyadic relation Suce, and the following rules
Even(X) + Even(Y) A (Y,
Even(0) <- Zero(O)
Succ2 X) + (2, A Suco Z)
While inventing such auxiliary predicates substantially reduces hypotheses (Cropper & Dumančid
2022] and improves learning performance (Cropper] (209),, predicate invention is expensive, inac- curate, and restricted low-arity invention (Cropper & Morel 2021b).
4.2 INSUFFICIENT NOISE HANDLING
Mislabeled & ambiguous data Noise is ubiquitous in realistic data. While symbolic methods struggle to learn from noisy data (Hocquette et al. 2024). neuro-symbolic methods can deal with mislaheled examples (Glanois et al.] (0022) and ambiguous BK (Evans & Grefenstette 2018). Han-
dling mislabeled BK is still an open problem (Cropper & Dumančié 2022).
Open-world assumption While the cloled-world assumption (CWA) asserts any ground atom, whose predicate appears in the BK, to be false if it is not given in BK, the open-world assumption (OWA) allows those ground atoms that are not known to be true to have the possibility of being true (Reiter 198T). Almost all existing IIP systems assume CWA. However, OWA is alistic setup since complete BK is inaccessible in real-world applications. A set of incomplete background ground atoms is considered noisy if ILP system assumes CWA.
4.3 MULTI-TASK LEARNING
Existing ILP focuses on once learning one target predicate. However, simultancously learning sev- eral target predicates can share common rules and capture mutual recursions among target predi- cates. Glanois et al. (2022) proposes an iterative multi-task learning scheme for their neuro-symbolic model and successfully learns certain hypotheses at a small program size. Large-scale multi-task learning is promising direction for building broader applications, which remains unexplored.
4.4 UNABLE TO LEARN FROM RAW INPUT
One major gap between II.P and modern MI. systems the ability to induce knowledge from raw sensory input. Most ILP systems only receive symbolic data as input, while raw data is usually images, speech, natural language, etc. There are initial works that use neural networks to perceive and transform raw input into symbolic form and use symbolic deduction to do reasoning (Manhaevel et al. 2018 Dai et al. 2019: Dai & Muggleton 2021). Efficiently jointly training perception network and ILP system in large-scale scale tasks would be promising direction for robust and reliable artificial intelligence. See Evans 2000);; Evans et al. 2021 2022) for more possibilities to induce rules from raw sensory input.
5

---

Published as a conference paper at ICLR 2025
Table 1: Predicates used in GeolLP (Target=can be used target predicate; Head=can be used in the head; Body=can be used in the body). Details see Appendix A.1
Name
Arity
Target
Argument Symmetry
Constraints
Head
Body
Name
Coll
collinear
Ncoll
X
not collinear same side midpoint segment congruent
Sameside
X
/
X
Midp
/
V
/
/
Cong
V
Perp
V
V
V
V
perpendicular parallel concyelic circle equal angle equal ratio
Para
V
Cyclic
J
Circle
Eqangle
Eqratio
5 GEOILP
In this section, we formally introduce GeolLP, our proposed dataset for elaborating large-scale so- phisticated rule induction systems, in four parts. First, a general guide for synthesizing ILP tasks from predefined rules. Second, the steps for identifying the examples (deduction step) and the BK (traceback step) GeolLP. Third, the features of GeolLP critically differ from other ILP testbeds. Fourth, the approach to transform symbolic input into raw sensory input.
5.1 A GENERAL GUIDE FOR ILP TASK SYNTHESIS
Inductive reasoning can be seen as a "reverse" procedure of deductive reasoning in the sense that the former learns rules from premises and conclusions while the latter infers conclusions from premises and valid rules. In ILP, premises correspond to the BK, and the conclusions correspond to the target examples. Therefore, construct ILP data, we can derive target examples from selected premises and predefined rules using a deduction engine.
Concretely, the dath-sytt procedure works follows:
1. Randomly intentionally choose several ground atoms as premises.
2. Define a consistent set of rules.
3. Deduce all the conclusions from the premises and rules using any deduction engine.
4. Select part or all the conclusions (also ground atoms) as positive examples. F
5. Trace back from the conclusions identify minimum of premises contributing to deducing the conclusions as ILP BK.
6. Optionally, obtain negative examples by removing the conclusions from syntactically possible ground atoms (all combinations of target predicate and constants).
5.2 SYNTHESIZING GEOILP
We choose plane geometry the application domain its formal system covers all the difficulties indicated in section 4 (see section 5.3).
Formalizing plane geometry and building corresponding symbolic deduction engine challeng- ing works outside this paper's 's scope. We adopt an expert-designed deduction engine based on deductive database theory (Gallaire et al. 1984). similar to the ones used in automated geometry
2 A of rules is consistent if the rules do contradict each other.
"Merely selecting a part of the conclusions as positive examples may yield fewer rules with more body atoms. P1 Pz A Q and Pz +- R, A R2 may he reduced to P, +- Rí A R2 A Q if the premises involve Q R1, R2 and only select P1 conclusions.

---

Published as a conference paper at ICLR 2025
Cong(O,B,0,C)
Per
Mi,
Perp(0,,0,8,C
Col1(D,0,H)
Cyc
Nco11(A,C,0)
H
Perp(E,C,C,0)
Espngle(CC.5,,,0AAA...A4
Perp(E,A,A,0)
Cyclic(O,AAD,C)
(C,,))
(ong(O,A,0,,)
Egpnglo(CG,,C,.000.44...0000
Premises
Deduction Traceback
Conclusions
Figure 1: Synthesizing one of GeolLP (single) tasks from one set of premises with Cyclic as target predicate. Black arrows denote deduction (forward chaining), and red arrows denote traceback.
theorem proving (c.g.. GEX (Chou et al. 2000). JGEX YYe et al. 2010a bj 2011). AlphaGicometry (Trinh al. ). The only constants the points in the plane. The engine leverages set of Horn rules for deduction. T shows the characteristics of predicates.
Deduction initialize deduction step with premises, we use plane geometry problems given in JGEX (Ye et al. [2010a]b] 201T), ranging from textbook exercises, regional olympiads, and famous theorems. An example of premises is depicted in Figure The final dataset, built from such se- lected premises, can effectively help construct automated geometry theorem provers without needing expert-defined rules, as the rules learned by ILP can be useful. Then, the deduction engine uses for- ward chaining reach the fix point. In the single-task setting, separate conclusions with different predicates.
Note that the deduction engine regards substantially reduces deduction costs since argument symmetry is omnipresent. Therefore, rules for argument permutation and several trivial rules are explicitly listed. See Appendix A.4 for details.
equivalent atoms as the same, which
Traceback Since the conclusions deduced from set of premises may involve all 8 predicates, synthesizing single ILP task should filter out those premises irrelevant to the target predicate. To achieve this, the deduction engine constructs deduction graph when doing forward chaining, which illustrates the immediate dependence the ground atoms in the graph. Every body atom in matched (ground) rule has a directed edge pointing the head atom (see Figure []. Starting from the conclusions involving only the target predicate, we trace back along the directed edges in the reverse direction until reaching the premises. The braked premises are regarded as the BK. The directed edges alongside (red arrows in Figure T constitute reference hypothesis.
After deduction and traceback, we repeat the BK and target examples ten times, retaining predicates unchanged but mapping every point to new, unique points. In other words, the initial group of points duplicated into ten groups. Then, the data are divided into training set and evaluation set according to 8:2 point groups.
5.3 DATASET FEATURES
In total, 63 expert-defined rules and many more trivial rules encoded in the deductive database are used for synthesizing GeolLP (see Appendix A.4). Several rules are listed here to demonstrate how
Note that GEX, JGEX, and AlphaGeemeety deductive reasoning algorithms and are not applicable to ILP. which is inductive reasoning.

---

Published as a conference paper at ICLR 2025
Table Detailed comparison between GeolLP and the existing dataset. (# denotes number of)
Dataset
OILP
GeolLP
835 single-tasks:
#Tasks
20 single-tasks
207 multi-tasks
Predicate Arily
1-2
--8
Prodicate Invention
60%
96% for single-tasks
Max #Variables
6
12
Max #Body Atoms
2
9
10-100 for 85% single-tasks
and 50% multi-tasks; S 1783 for single-tasks;
#Rules
1
S 3553 for multi-tasks
in every predicate:
in just a few predicates; limited to dyadic predicates
Argument Symmetry
various symmetries in up to ogdoadic predicates
Predicate Constraints
a few predicates
in every predicate
linear recursion: 30%
recursion: ~ 100%
Recursion
non-lincar recursion: 10%
mutual recursion: ~ 90%
lincar mutual recursion: 10%
all 4 kinds of recursion common
GeolLP covers the limitations mentioned in section 4
(1) Midp(M, C, D) + Midp AM, A, B) A Para Para(A, A Para(A,D,,,C
(2) Para(A, B, E, F) + Para( A, B, C, D) Para(C.D,E,F)
(3) Cong(A, M. B, .) + Perp( A, B, B, C) A C,))
(4) Cong(O, A, O, B) + Midp( M, A, B) A Perp(O, A, B)
(5) Perp B, P. Q) + Cong(A,, B, P) A Cong(A, (A, Q, B,Q)
We briefly introduce the meaning of the predicates appearing in the above rules and Figure I to facilitate understanding them. See Appendix A. for exhaustive descriptions. Midp A, B) asserts M is the midpoint of segment AB. Para(A, B, C) asserts lines .AB & CD are parallel. Perp(A, A, B, C, D) asserts lines AB CD are perpendicular. Cong(A, B, B,C. C, D) as- serts segments AB & CD are of same length. (A, B, C) asserts A, B. are collinear. Ncoll(A, C) asserts A, B, C' are not collinear. Cyclic(A, (A, B, C, D) asserts A, B, C, D are con- cyclic. Eqangle(A) (A, B, C, F, G, asserts full-angles [AB, CD) & [EF, GH) are equal. Full- angle is defined by two lines and, intuitively, two full-angles [AB, CD] & &EF, GH) are equal if, supposing Rot denotes rotation, Rot(AB) || EF and (ot(EF) GH. Refer to Ye et al. (010b) for the formal definition.
Predicate Characteristics of predicates are provided in Table The predicates are of arity from to 8, all involving argument symmetry (see Appendix [A.2] and different constraints (see Appendix A.3). Note that, among constraints, Midp and Circle are functional; thus, our dataset can easily adapted to study the setting with functions. In addition, nearly all the predicates can be used in the head and body of a rule.
Rules The number of rules in 85% single-tasks ranges from at least 10 to 100, while the rest of single-tasks involve hundreds to thousands of rules. These rules result from expert-defined rules, argument symmetry rules, and trivial rules. See Appendix A.4 for details about trivial rules. The maximum number of body atoms is 9. The maximum number of variables in rule is 12. Exis- tentially quantified variables usually exist. For instance, A, B of Rule(1), C, D of Rule(2), C of Rule M of Rule(4).
8

---

Published as a conference paper at ICLR 2025
Recursion All types of recursion omnipresent in the whole rule set, and almost any two pred- icates are mutually recursive. In the example rule subset above, Rule(1) is recursive and linear; Rule(2) is recursive but not linear, (3)e(3) & Rule(5) (or Rule(4) & Rule(5)) justify that they recursive because Cong and Perp are mutually recursive; while Rule(3) (also Rule(4) is linear, (5)e(5) is not linear.
Predicate invention Almost all GeolLP tasks require predicate invention. For example, task learning Para with Cong, Coll, Ncoll, Eqangle, Perp may need to invent Midp as auxiliary predi- cate (Appendix B.T). In addition, extra meaningful relations that are not used in GeolI. may also be invented to reduce hypothesis space, e.g., congruent triangle, similar triangle (Appendix B.
Noisy data First, our synthetic data makes the open- world assumption. For instance, in Figure I Perp appears twice in the BK (premises), while two new atoms of Perp not given in the BK also appear in the deduced conclusions, which means that atoms not given in the BK can also be true. In GeoILP, it is common for all true atoms not to be given in BK. Second, the negative examples are noisy since our rule set is incomplete for the entire plane geometry (i.e., several true atoms may not be deduced based on the incomplete rule set).
Multi-task the multi-task setting, we trace back premises from all conclusions; thus, all predi- cates in the deduction graph are considered target predicates for an ILP multi-task. In most cases, the trace-backed premises are the same as the initial premises.
5.3.1 COMPARISON WITH OTHER ILP DATASETS
We compare GeolLP with the dataset proposed in öILP (Evans Grefenstette 2018). which is the only synthetic dataset used by recent neuro-symbolic methods (e.g., HRI [20]). Table reveals GeolLP's extremely strong complexity from various aspects.
5.4 CONSTRUCTING RAW INPUT
An essential difference distinguishing it from other datasets that GeolLP additionally provides raw inputs corresponding to each task. Like in Figure D the BK (premises) is transformed into an image like in the plane geometry textbook. We adopt the constructive diagram builder language developed in AlphaGeometry (Trinh et al.) (0224) to construct the image point by point from a given set of premises, which works well with the symbolic deduction engine. The goal to provide data for learning rules from raw sub-symbolic inputs (images) and symbolic target examples. The images only contain basic geometry objects, reducing the burden of perception and making them a good testbed for this immature research topic. We also attach corresponding image of conclusions (BK + positive examples) to each task, like the rightmost diagram Figure [ Handling GeolLP in geometric form at least requires the ability to identify geometric objects, identify the relations among objects, and to induce interpretable rules. Developing such a complex system requires great effort, which is out of the scope of dataset constructing work and is left future work. g
More details about datasets and example data are provided in Appendix & B respectively.
6 EXPERIMENTS
6.1 SETUP
Considering the great difficulty of GeolLP, we divide into four progressive levels, which provides chances for gradually strengthening ILP systems. Table 3 fillussrates the specification of each level.
Hence, multi-tasks do not involve predicate invention.
'The numbers of rules are lower bounds of actual numbers, since it is hard to count all trivial rules and argument symmetry rules that are implicitly encoded in the deduction engine.
Thhese statistics from our provided reference hypotheses. An II.P system may learn other possible hypotheses with different statistics.
'For readers interested in the induction ability of large language models, we provide guide on how to translate GeolL. .P into natural-language form in Appendix [

---

Published as a conference paper at ICLR 2025
Table 3: GeolLPPs four single-task levels & multi-task level.
Level
basic
simple
advanced
complex
multi-task
(single) tasks%
2%
9%
23%
66%
arity # body atoms
S 8
S 8
I 8
S 8
S 2
S 9
5
9
< 9
mutual recursion? # rules
X
20
S 20
S 20
S 1783
S 3553
The four levels are set according four dimensionalities: predicate arity, number of body atoms, involving mutual recursion or not, and number of rules. The former three are critical bottlenecks of many neuro-symbolic ILP methods, e.g., Glanois (2022). The last one is critical aspect affecting search complexity for both symbolic and neuro-symbolic IIP methods.
6.2 RESULTS
Symbolic Among symbolic methods, Popper (Cropper Morel 2021a) is the most powerful one that simultancously supports learning recursion, involving hypothesis constraints, inventing predi- cates (Cropper & Morel, 20216), and handling noise (Hocquette al.] 2024). and scales better as well. We conduct experiments using Popper |
enabling predicate invention, recursion and noise handling. Noise handling turned on because GeolLP follows OWA, while Popper follows CWA. The maximum number of variables in rule is set to 12, which is the maximum value in every four levels. When conducting experiments on different levels, we set the maximum number of body atoms and the maximum number of rules to the maximum values of the learning level. This setup aligns with our purpose of not injecting many priors into training. After I-day searching, Popper does not return any hypothesis, since the searching time already about two orders of magnitude longer than in previous works (about hundreds of seconds or less, e.g.. Ce.ppprrrrr & Morel (2021a) Glanois et al. 2022)
the basic level. We regard GeolIPP unsolvable by Popper
Neuro-symbolic Several neuro-symbolic methods (Evans & Grefenstette 2018 Glanois et al, 2022) do not support higher-arity predicates, rules with more than two body atoms, or using target predicate the rule's body, which are their primary bottlenecks of being inapplicable to GeolLP. Several others (Rocktäschel & Riedel 2017: Campero al. 2018) require expert-defined rule tem- plates task-by-task, which inappropriate for large-scale applications like GeolLP. Difflog (Si et al.] 2019) is neural-symbolic method that supports arbitrary hypothesis space. However, ments show that, even at the hasic level, Difflog TT throws an out-of-memory error on server with 500GB of memory, an order of magnitude larger than in the original paper (64GB), We leave further investigation on improving memory usage for future work.
experi-
DISCUSSION AND CONCLUSION
We propose GeolLP, a large-scale synthetic dataset for inductive logic programming involving all challenging language biases reference hypotheses. GeoILP is, in terms of the hypothesis size, at least one magnitude larger than existing datasets that provide guiding hypotheses. Although GeolLP may be biased towards plane geometry, it is still a good testbed for large-scale ILP. Be- sides, also provide image-form background knowledge, aiming boost the development of joint learning of neural perception and symbolic rule induction.
9 Version 4.3.0: htt.:////....tooooooooooo///aaaa
thub.
WeWe leverage the implementation and recommend parameter setting in https://githh........ htt. aac
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported by the National Natural Science Foundation of China (No. U23B2056), in part by the Fundamental Research Funds for the Central Universities, and in part by the State Key Laboratory of Complex & Critical Software Environment.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303. 087744, 2023.
Francois Bancilhon and Raghu Ramakrishnan. An amateur's introduction to recursive query pro- cessing strategies. In Proceedings of the 1986 ACM SIGMOD international conference on Man- agemenl of data, pp. 16-52, 1986.
Sándor Bartha and James Cheney. Towards meta-interpretive learning of programming language semantics. In Inductive Logic Programming: 29th International Conference, ILP 2019, Plovdiv, Bulgaria, September 3-5, 2019, Proceedings 29, pp. 16-25. Springer, 2020.
Hendrik Blockeel and Luc De Raedt. Lookahead and discretization in ilp. In International Confer- ence on Inductive Logic Programming, pp. 77-84. Springer, 1997.
David A Bohan, Corinne Vacher, Alireza Tamaddoni-Nezhad, Alan Raybould, Alex Dumbrell, and Guy Woodward. Next-generation global biomonitoring: large-scale, automated reconstruction of ecological networks. Trends in ecology & evolution, 32(7):4774877, 2017.
Antoine Bordes, Nicolas Usunier, Alberto
Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. Advances neural information pro- cessing systems, 26, 2013.
Andres Campero, Aldo Pareja, Tim Klinger, Josh Tenenbaum, and Sebastian Riedel. Logical rule induction and theory learning using neural theorem proving. arXiv preprint arXiv:1809.02193, 2018.
Shang- Ching Chou, Xiao-Shan Gao, and automated geometry theorem proving and discovering. Journal of Automated Reasoning. 25(3): 219-246, 2000.
Zhang. A deductive database approach to
Andrew Cropper. 2019.
Playgol: Learning programs through play. arXiv preprint (XXXv:1904.08900
Andrew Cropper and Sebastijan Dumančić. Inductive logic programming 30: new introduction. Journal of Artificial Intelligence Research, 74:765-850, 2022.
Andrew Cropper and Céline Hocquette. Learning logic programs by discovering where not to search. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 6289-62900 2023.
Andrew Cropper and Rolf Morel. Learning programs by learning from failures. Machine Learning, 110(4):801-85,, 202
Andrew Cropper and Rolf Morel. Predicate invention by learning from failures. arXiv preprint arXiv:2104 /4426, 2021b.
Andrew Cropper, Sebastijan Dumančić, and Stephen H. Muggleton. Turning 30: New ideas in inductive logic programming. In Christian Bessiere (ed.), Proceedings of the Twenty-Ninth Inter- national Joint Conference on Artificial Intelligence, IJCAI-20, pp. 4833-4839, International Joint Conferences on Artificial Intelligence Organization, 7 2020a. doi: 2t.22463/jcaii htttps://oii.oo
URI.
49449/3/20 124433/ijcai..... http://caaaa.../...... Survey track.
Andrew Cropper. Richard Evans, and Mark Law. Inductive general game playing. Machine Leam- ing, 109: 109:1393-1434, 2020b.
11

---

Published as a conference paper at ICLR 2025
Wang Dai and Stephen Muggleton. Abductive knowledge induction from raw data. In Zhi-Hua Zhou (ed.), Proceedings of the Thirtieth International Joins Conference on Artificial Intelligence, IJCAI-21, pp. 1845-1851. International Joint Conferences Artificial Intelligence Organiza- doi: (0.24966/ijcai.22222222 URL https:///o/t... http:///do...r110.2222222 jca.. jcai.
tion, 8 2021.
202 //22//54
Main Track.
Wang- Zhou Dai, Qiuling Xu, Yang Yu. and Zhi-Hua Zhou. Bridging machine learning and logical reasoning by abductive learning. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL http:::://pppoceedinneeo neur cc/pper.r fi w1les/paper/2019//// http:////....aoo.e.eeeea.......a pape
Honghua Dong. Jiayuan Mao, Tian Lin, Chong Wang, Lihong Li, and Denny Zhou. Neural logic machines. In 7th International Conference on Learning Representations, ICLR 2019, New Or- leans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL Forum -ind-iidBBxx bRctX
Richard Evans. Kant's cognitive architecture. PhD thesis, Imperial College London, 2020.
Richard Evans and Edward Grefenstette. Learning explanatory rules from noisy data. Journal of Artificial Intelligence Research, 61:1-64, 2018.
Richard Evans, José Hernández-Oralle Johannes Welbl, Pushmeet Kohli, and Marek Sergot. Mak- ing sense of sensory input. Artificial Intelligence, 293:103438, 2021. ISSN 0004-3702. doi:
https://doii.or/100.000 http:://rrrr/.11111oooooooooam0m artint artp::....rrimt..220.0. 2020. URL https::///// //ttp:://ww/ www.eccdireecttt...cccc..e.eeeee wttppp////w...cceedddrreeee com/
Sellence/arrticle/p11////
Richard Evans, Matko Bošnjak, Lars Buesing, Kevin Ellis, David Pfau, Pushmeet Kohli, and Marek Sergot. Making sense of raw input (extended abstract). In Lud De Raedt (ed.), Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22, pp. 5727-5731. International Joint Conferences on Artificial Intelligence Organization, 7 2022. doi: 10.24963/ ijcai 2022/799 URL BRLphtts::///om ttgp:/io.......ooo 24963/1jcai wtt.:/....................... 799 Journal Track.
Herve Gallaire, Jack Minker, and Jean-Marie Nicolas. Logic and databases: A deductive approach. ACM Computing Surveys (CSUR), 16(2):153-185, 1984.
Claire Glanois, Zhaohui Jiang, Xuening Feng, Paul Weng, Matthieu Zimmer, Dong Li, Wulong Liu, and Jianye Hao. Neuro-symbolic hierarchical rule induction. International Conference on Muchine Learning, pp. 7583-7615. PMLR, 2022.
Céline Hocquette, Andreas Niskanen, Matti Järvisalo, and Andrew Cropper. Learning mdl logic programs from noisy data. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 10553-10561, 2024.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6700-6709 2019.
Katsumi Inoue, Andrei Doncescu, and Hidetomo Nabeshima. Completing causal networks by meta- level abduction. Machine learning, 91(2):239-271 2013.
Samina Khalid, Tehmina Khalil, and Shamila Nasreen. A survey of feature selection and feature extraction techniques in machine learning. In 2014 science and information conference, pp. 372- 378. IEEE, 2014.
Mark Law, Alessandra Russo, and Krysia Broda. Inductive learning of answer set programs from noisy examples. arXiv preprint arXiv: /808 08441, 2018.
Mark Law, Alessandra Russo, Elisa Bertino, Krysia Broda, and Jorge Lobo. Representing and learning grammars in answer programming. Proceedings of the AAAI Conference Artificial Intelligence, 33(01):2919 2928, Jul. 2019. doi: 10.1609/amai. 33300..33110101111133333 URL hetps:
7/o aaai www.oogiioo. wnw.://ndexx..cm http:///ieeeepppp////aaaaaaaaaa.
12

---

Published as a conference paper at ICLR 2025
Mark Law, Alessandra Russo, and Krysia Broda. The ilasp system for inductive learning of answer set programs. arXiv preprint arXiv:2005.00904, 2020.
Robin Manhaeve, Sebastijan Dumancie, Angelika Kimmig, Thomas Demeester, and Luc De Raedt. Deepproblog: Neural probabilistic logic programming. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (cds.), Ad- vances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL hutpp:://prooceedansss.eeeuxuuop httpa//cee...aaoo.oooo/ooooeooer Flle:/ccad777ceee222363333363333 Papeer.aaapprrpppp.p
Stephen Muggleton. Inverse entailment and progol. New generation computing, 13:245-2866 1995.
Stephen Muggleton and Luc De Raedt. Inductive logic programming: Theory and methods. The Journal of Logic Programming, 19:629-679, 1994.
Stephen Muggleton, Lue De Raedt, David Poole, Ivan Bratko, Peter Flach, Katsumi Inoue, and Ashwin Srinivasan. 1pp turns 20: biography and future challenges. Machine learning, 86:3-23, 2012.
Stephen H Muggleton and Christopher H Bryant. Theory completion using inverse entailment. In International conference on inductive logic programming, pp. 130-146. Springer, 2000.
Stephen H Muggleton, Dianhuan Lin, Niels Pahlavi, and Alireza Tamaddoni Mamaddoni-Nezharre -Nezhad. Meta- interpretive learning: application to grammatical inference. Machine learning, 94:25 2014.
Stephen H Muggleton, Dianhuan Lin, and Alireza Hamaddoni-Nezhanne Meta-interpretive learning of higher-order dyadic datalog: Predicate invention revisited. Machine Learning, (0:41)) 2015.
J. Ross Quinlan. Learning logical definitions from relations. Machine learning, 5:239-266, 1990.
Raymond Reiter. On closed world data bases. In Bonnie Lynn Webber and Nils J. Nilsson (eds.), Readings in Artificial Intelligence, pp. 119-140. Morgan Kaufmann, 1981. ISBN 978- 0-934613-03-3.
doi: http::/co..ooyo/o.00009999444441
URL https:
//wwws www. www.:cacceerrrcccccoooo......aaa
Tim Rocktäschel and Sebastian Riedel. End-to-end differentiable proving. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://ppoceeeings..ee..... 2at.:6000066e-----.....---- pdf
cctpappr.prpeeeeeeeeeeeee files/paper/2017/
Stuart Russell. Human compatible: Al and the problem of control. Penguin Uk, 2019.
Xujie Si, Mukund Raghothaman, Kihong Heo, and Mayur Naik. Synthesizing datalog programs using numerical relaxation. In Proceedings of the Twenty-Eight International Joint Conference on Artificial Intelligence, IJCAI-19, pp. 6117-6124, International Joint Conferences on Artificial Intelligence Organization, 7 2019. doi: 0.249633/jcai.20199833 URL htt.hhttt:///tt //ctps:://oi... orgg/ 10.24963/1Jcai 2019//47
Aishwarya Sivaraman, Tianyi Zhang, Guy Van den Broeck, and Miryung Kim. Active inductive logic programming for code search. In 2019 IEEE/ACM 41st International Conference on Soft- ware Engineering (ICSE), pp. 292-303, IEEE, 2019.
SRINIVASAN. The aleph manual. http://www..coccoccdcccccccoococ 2001.
Alireza Ramaddoni-Nezhhee Raphael Chaleil, Antonis Kakas, and Stephen Muggleton. Application of abductive ilp to learning metabolic network inhibition from temporal data. Machine Learning, 64:209-230. 2006.
13

---

Published as a conference paper at ICLR 2025
Kristina Toutanova and Danqi Chen. Observed versus latent features for knowledge base and text inference. In Alexandre Allauzen, Edward Grefenstette, Karl Moritz Hermann, Hugo Larochelle, and Scott Wen-tau Yih (eds.), Proceedings of the 3rd Workshop Continuous Vector Space Models and their Compositionaliyy pp. 57-66, Beijing, China, July 2015. Association for Com- URL
putational Linguisties. doi: 10. 118855/11/11554444
W15-4007
Trieu H Trinh, Yuhuai Wu, Quoe V Le, He He, and Thang Luong. Solving olympiad geometry without human demonstrations. Nature, 625(7995):476-482 2024.
Fan Yang, Zhilin Yang, and William Cohen. Differentiable learning of logical rules for knowledge base reasoning. Advances in neural information processing systems, 30, 2017.
Zheng Ye, Shang-Ching Chou, and Xiao-Shan Gao. Visually dynamic presentation of proofs in plane geometry: Part 1. basic features and the manual input method. Journal of Automated Reasoning, 45:213-241, 2010a.
Zheng Ye, Shang-CChing Chou, and Xiao-Shan Gao. Visually dynamic presentation of proofs in plane geometry: Part 2. automated generation of visually dynamic presentations with the full- angle method and the deductive database method. Journal of Automated Reasoning, 45:243-266, 2010b.
Zheng Ye, Shang-Ching Chou, and Xiao-Shan Gao. An introduction java geometry expert. In Automated Deduction in Geometry: 7th International Workshop, ADG 2008, Shanghai, China, September 22-24, 2008. Revised Papers 7, pp. 189-195. Springer, 2011.
Zheng Zhang, Levent Yilmaz, and Bo Liu. critical review of inductive logic programming tech- niques for explainable ai. IEEE Transactions on Neural Networks and Learning Systems, 2023.
14

---

Published as a conference paper at ICLR 2025
A DETAILS OF GEOILP
A.1 PREDICATES DESCRIPTION
In this section, we exhaustively describe the predicates.
The constants in GeolLP are points in the plane. The exact meaning of each predicate is listed below.
MM, A, B) asserts M is the midpoint of segment AB.
Para(A, B, C, D) asserts line AB and line CD are parallel.
Perp(A, B, C, D) asserts line AB and line CD are perpendicular.
Cong(A, B, C, D) asserts segment AB and segment CD are of same length.
Coll(A, B, C) asserts A, B, C collinear.
Ncoll(A, B, C) asserts A, B, C are not collinear.
Cyclic(A,, B, C, D) asserts A, B, C, D concyclic.
Eqangle B,
F, G, H) asserts full-angles [AB, CD) & [EF,GH] are equal.
Full-angle is defined by two lines. Intuitively, two full-angles [AB, CD) & (EF, GH are equal if, supposing Rot denotes rotation, Rot(AB) EF and RoD(CD) GH.
Refer to Ye et al. (210b) for the formal definition.
Eqratio(A, B. C, D, E, F, G. H) asserts the ratio of segment AB and segment CDD equals to the ratio of segment EF and GH.
Circle(O,, A, B, C) asserts A, B, C are concyclic and the center is O.
Sameside(B, A, C,Y, Z) asserts B is to the same side of A&C Y is X&Z. Math- ematically, this is equivalent (BA BC) YZ) 0.
Sameside are not allowed as target predicates because it does not appear in the head of any rules. Coll and Ncoll are not allowed as target predicates because the rules where they appear the head are either too few or trivial (see Appendix A.4 for trivial rules).
A.2 ARGUMENT SYMMETRY PREDICATES
We use biconditional to describe the argument symmetry of predicates.
Midp(M,, A, B) ++ Midp( A)
Para(A, B, C, D) ++ Para D, B)
Para(A,B, C, D) ++ Para(B,A,
Perp(A, D)
(e,,CC,,,A,B Perp(A, C, D) Verp(B,A,C,
Cong(A, C,D) +*
Cong(A, C,D) +* A,
Coll(A, C) Coll(A,C,B)
Coll(A, B, C) ++ Coll(B,C,A)
Ncoll(A, B, C) + Ncoll(A, C, B)
NcolI(A, B, C) ++ Nco
Cyclic(A, B, C, D) + Cyclic B, D,
Cyclic(A, B, C, D) Cyelic B, A, A,,
Circle(O, A, B, C) ++ Circle(O, (O,,d,,,,CCAA
Circle( O, A, B, C) Circle(O,A,C.
Sameside B, A, C, Y, X, Z) Sameside X,Z,B,ACC
15

---

Published as a conference paper at ICLR 2025
3amcside(B.A......X.3)) Samesiee(B.CAA....2] Cyangle(A,B,C,,D,..,, G, H) + Enengle(E.F,,.HHAAA.. A, + EqangleeC..DAA F
A,AnnleeeAA.....D.EEEEGG.. Eqangle(2,,A,C,,,,,
Eqratio (A, A, + Eqratio(EE..,,,HAAAABBBC Eqratio(A, A. + C.nntoo(.DD.AABG.HHEE...
B,,,
+
Note that these biconditional formulae constitute minimal representation of argument symmetry. Much more symmetries can be derived by combining several of them.
The rule set would be too large if these argument symmetries all represented Horn rules. A compact representation and effective handling should be thus considered by ILP methods.
A.3 CONSTRAINTS OF PREDICATES
We describe the constraints of predicates. We express the constraints by Horn goal, as Cropper & Hocquette 2023).
Sameside C,, A Sameside(A, B, A, C,,,
Midp(M,, A, B) A Midp(N, A, B) M #
Circle(O, A, B, C) A Circle A, A,
+
Perp(A, B, C, D) A Perp(C, D, F) A B,
Eqangle B, E,
A
Eqangle B, A
Npora(A, B, C, D)
Eqangle(A, A, B, E, A C. A Ncomg(A,B,C.D)
+
(Npara for not parallel; Ncong for not congruent)
A.4 RULES
All the Horn rules utilized to synthesize GeolLP are divided into expert-defined rules, trivial rules, and argument-symmetry rules (if representing argument symmetries by Horn rules).
Trivial rules are Horn rules trivial for human. For example, Coll(A, B, < Coll(A, B) A Coll(A, D, C). Note that, though trivial for human, ILP learner may suffer from the large number of trivial rules. A whole set of such trivial rules is illustrated in Chou et al. 20000 and is encoded into the deductive database.
The expert-defined rules are
1. Pa < Pe C, P.p A Ncoll( B,
2. Cyclic(A, B,
+ Cong 0, A, 0, A Cong B, B, A Co C) C) B,
3. Para(A, C, + Cqp.GllAA.BBP.Q.C..DD...
4.
Q B,
5.
A, < PP AP A Neoll(P,
6.
Cyclic(A, A, B, C, Cyclic(A, C) A Cyclic(A, A)
7. Fara(E,F,B,C) <- Midp(E,ddM)))... A, B) A Midp
8. Eqratio3(A, C, D,0,0) Para(A, B,C, C,D) Coll(O, A, C) Coll(O, B,
9. Eqangle G. Perp(A, B,C.D) D) Perp F,, F,G.
10.
Eqangle (A, C, ^
Eqangle C, F,
16

---

Published as a conference paper at ICLR 2025
11. Eqratio BA, E, F.
Egrato(A.B.C.DDMM...R A
( E, C.P.Q.QQQRRR
Eqangle(A, B, A, D,A, + A.g.:.....B.....A.AAAAAA
B, 13. Eqratio(D, B, D, B, A,C) - Eqangle (A, B, A, D,
NcolI(A,
14. Eqangle(O,A, A, B,A,B,O,B) Cong(O, A,A,, B) A Neo (,,,( < O, B, A, B, 0) A Neoll(O, (,, A, + Circle(O, A, B, C) Perp(O,A,,..)
16. Eqanglo(A.XXAA.BBC.AA
17. Ae, - Circle
Eqangle (A, X, A, B, C,A,C,
18. Cypngle((A,B,A,C,,.....
Circle(O, A, B, C) Mi,
19. Midp(M,B,C)
A.B,CC
Coll(M, B,C)
A
Eqangle(A, B, A, B,
20. CA, B, t Perp(A, A, B, C) Mid
21. BB) + Circle( B,C, Coll(O,,AC))
22.
+ Cyclic( A, B,C,, PAra(A,
23.
+ Midp( M, A, B) A (r,
24. Porp(A, - Cong(A, P) A (,,
25. Porp(P,A,A,,)) Con Cong (A,Q, B,Q) Cyclic(A, B, P.Q)
26. Para(A,C,B,D) (alp:)))MMdd(MMMM..M Midp(M,, B3 Midp(M,,,D)
27. Midp(M,C,,D) MMM
Midp((M, A, B) A Para(A, C, B, Para A, D. D,
28. D) Eqratio(O) (O, A, A, B, B,
A, A,CACll,, A)A
Ncoll(A, (,, A Sameside( A, C2 D,
29. C, C) + Para(A, A,
30. Eqratio A, B, C, + Midp(M, A, B) A Midp(N,
31. (er D) + Eqangle(A A, B, P. D,U,V, A Perp(P,QQU,.)
32. C, D) < Eqratio (A, B, P,Q.C, Cong( V),
33. Eqangle(A, A, P.Q. Cong(C,A,FF, (A,
+
A Cong(B,C,Qu,
34. A, C, P,Q,
(A, Q, ^ (ong(B,C,o) B,C,Q, ^
Ncoll (A,
35. C, C, P.R.C. Q) Cong(C,A,RR,)) Ncol
Cong B, A
36. Cong(A,C,P,I) + Cong(A, (Ang(A B. P,Q) ^ Cong(B, BR, ^
A Ncoll A, B,C.
37.
Cong( B, A Cong(B,C,Q) (B,C Ncoll(A,, C)
38.
+
Cong( A, B,P.P.Q A Cong(B,C,Q,, ^
39. TcpangleAA.BAAACCP...0) A Ncoll(A,, B,C)
Eqangle B, B,
(TAann((C.AAC,,,,RRPPP..... Ncoll(A, B,C)
40.
+
Eqangle BB ^
C,A,, A Ncoll B,C)
41. Eaqaaio(1B.C.,,,,,CC R,B ^ Ncoll(A, B,C)
Eqangle (B.AABBC.QQP..OOII
+ 42. Eqangle (C, PC,.P. A Ncoll B,C)
BBangll(BBAABB....PPp.
^
17

---

Published as a conference paper at ICLR 2025
43. Cong C,
Eqangle(B..A1) A,,, (Bangll(BBAABBB..Q......
A
A [coll(A, B,C) Cong (A,
44. Cong (A, L'aanle(C,,A,,, B, R, Q) ^ Neoll(A, B, A Cong (A, P.Q
Eqangle [B, A, C. PQQ.))
45. Eeguugle(AA..AAA..P....... A Neoll B, C) Cong (,
Eqangle A, B, P.Q.
A
Ecpuale((..A.C.B.B..P..
46.
Egangle B, A, B, C, Cq B, Q, Ncoll B, A B,
R,
A
47. Cong(B,C,Q,,) Eqaagle(C,,A,C, B, Q, R (A, B, B,C) A Cong (A, B, P.Q)
Egangle B, B, Q, Q,Q, R)Q,P)
A
48. Expngle(AA.BAAA...P.OO. CC..II,,CC..BB..ROO..... A (A, (, A Cong B,
Eqangle B, A, B, C. C,Q,
49. Ecanglo(A..BAA.....O..... B, R,Q) A Ncoll (A, A,
t
Eqratio (B.AAB,C.QQP.O...
50. (B,ratio(,,,C,,BB,, (A, (B. A
51. EqqatioC.AA.C...R..PE. (A,
Eqratio BB, (B..ABCC..Q...O... A
52. Eeengle(C..A...BBRR.PPP. Ncol(AA,
CqratioBBAA.BB..OO.P..... ^
53. (Ttengle(A.BBAACCPP.O... Eqangle(B.44, P... A AA B,C)
Eqratio C,
54. (rannlo(C..4,C.BBRR..I A Neoll(AA, B,C)
+
BBAA,,,,
55. (B,4,,,,,,,,,,,,,, NA B,C)
+
Eqratio B, A, B, ^
P,Q, ^ ( 56. BB
( B, BB A
B,
A, A,B,C)
57.
B, Eqratio(B.AAB,CC. B.A.B.CCO.P..O.R
C, B, R, P, R,Q) A Nool](A, (B,o)(A,,,))))) B,C) A Cong (,
58. Cong(A,C,P,I)) C,r,ato(C,,,,,,,,BR P, A Nc A Cong(A,
Egratio B, C,A,
59. (B,,atoo,.A.C,BBRRR., A Ncoll A, B, B)C) Cong (,
-
Eqratio (qratio(B,AA, B,A.
A
60. Exgangle(B,AABB.C.0..E CAP.atoO(CAAACCBBBRRRP.L. Ncoll (, B,C) Cong (A
-
A
61. BR.ngleeC.AACCBBBBRRR.... A Ncoll( B,C) Cong (A,) (qratio(C,A,CCBBBR.PP.
< Eqratin iorattoofo B, P.Q.) ^
62. N, Eqratio(M A,, M, D, B, A Sameside(MM (M,,, A, D,D,N, B,
Para(A, D. Coll(M, A, A Coll(N,B,C) A
63. Fqratio(M, M, D, N, C, Coll(N, B) A Paral (M, N, A,
Para (A, A,B.C, D) A Coll(M,
These rules adapted from those in AlphaGeometry.
httpps:/ thtp::://githtb...... com wtt.:/o..ooo///ggggggg -eepm
h jgex_ag._____
231
txt
18

---

Published as a conference paper at ICLR 2025
B
DATA EXAMPLES FROM GEOILP
B.1
PREDICATE INVENTION
The example below requires the invention of Eqangle
Background Knowledge:
Cong(D8.A8,D8.BB)
Cong(D8.B8,D8,
Cong(D8.E8.D8.A8)
Co11(C8.A8,F8)
Pexp(E8,F8.A8,CS,
A88)(G8,A88BB8)
Perp(E8,G8,A8,B3)
NC8.)(F8,G8.A8)
Cong(D9,A9,D9..B)
Cong(D9,B9,,9)
Cong(D9.E9,D9..4)
CoII(C9.A9.F9)
Fe99(E9.F9.A99.99
CoIl(G9,A9.B9)
Perp(E9, (9.
Nc0.)()9,G9,A9)
Positive Examples:
CyClic(C8,E8.A8.BB)
Cyclic(E8,A8.F8..
Cyclic(C9,F99AA,,H)
Cyclic(F9.A9,F9..C))
Proof steps:
Cyclic(C8.ES.A8.B5)
Cong(D8.A8.D8.BB)
Cong(D8.E8.D8.A8)
Dong(D8,B8,DD,.C))
CoII(F8,A8,C3) <- CoI1(C8,A8,F8)
CoII(G8,A88BB8) +- C88
Batp:/aAAA.....8.88........ G8)
Perp(E8.GG.AA.BBB) Perp(E8.F8.A8..8)
Eqangle(F8,A88 G8.8EE8,,G...... G8,
CoII(F8,A8,C3) A Co8I(G8,A8,B8) A Eqan-
Cyclic(EB,A88FF8.C + (ganga//B....F..B.GGG.......
(,8.)(F),G8.A88
Cyclic(C9,E9.A9..B))
< Cong(D9.A9.D9.B9)
Con9(D9.E9,,99.99)
Cong(D9,.9,.99.C9)
Co1I(F9.A9,C9) +- CoII(C9,A9,9))
CoII(G9.A9.B9) CoII(G9.A9.B9)
BganglcAA....EEBEPPPA.......... Perp(E9,69.A9..99) Perp(E9.F9.A9,.9)
Eqanglc(F9 FF..:/...AA.....B999.....0
+ CoIl(F9.A9,C9) A CoIl(G9.A9.B9) A Eqan-
Cyylic(E9
Nc091(99.G9.A9)
19

---

Published as a conference paper at ICLR 2025
NO
34
Figure 2: An example of forae-form background knowledge.
86
Figure 3: An example of image-form form background knowledge + positive examples.
B.2 RAW SENSORY DATA
See Figure 2 for example of image-form background knowledge. See Figure 3 for example of image-form conclusions, which is combination of background knowledge and positive examples.
C TRANSLATING GEOILP INTO NATURAL-LANGUAGE FORM
For readers interested in the induction ability of large language models (LLMs), we provide guidance for translating GeolLP into latural-language form.
The prompt fed to LLMs should consist of three parts: task description, task data, and a command requiring LLMs induce hypothesis. The difficult part is the task data, containing background knowledge and positive & negative examples, which are ground atoms. The translation from sym- bolic atoms natural- natural-language forms varies in different domains. For plane geometry, AlphaGeom-
20

---

Published as a conference paper at ICLR 2025
etry Trinh et al. 2024) provides several templates. For example, Coll(A, B, C) is translated into A,B, C are collinear. All the predicates used in GeolLP can be found in AlphaGeometry, and thus all GeolLP's atoms generating Horn rules (in natural-language forms), the command could ask LLMs to generate rules in the form of "If and ... then ... where each ".." corresponds to ground atom. "..." can be repeated multiple times and must be concatenated by "and".
be translated into natural- -lnguagg forms. In addition. to enforce LLMs
21


Reviews:
[{"review_id": "HH0SoCzzq7", "reviewer": "Reviewer_Y6Wf", "summary": "This paper introduces a large-scale synthetic dataset for inductive logic programming involving a number of challenging predicate language constructions.  The subject domain of geometry provides a rich set of predicates with symmetries, recursion, and constraints. Furthermore, when higher dimensional connective objects (e.g. line segments) are excluded from the background knowledge, predicates become black-box functions of the remaining universe of objects (e.g. points).  This setup provides an instructive setting in which to illustrate and investigate many prominent and unsolved challenges in ILP.", "questions": "The statement \"Enabling recursion is expensive for symbolic ILP, while neuro-symbolic ILP does not support mutual recursion\" cites a singe neuro-symbolic method that presumably does not support mutual recursion.  Is there a theorem that says that no neuro-symbolic method can support mutual recursion?  \n\nIs there a reason to exclude the wider neuro-symbolic techniques that have been explored?\n\nCan you discuss the limitations of binary predicates in more detail?  I think this deserves more attention.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- This paper provides a reference dataset for investigating a number of important challenges in ILP for which no current datasets exist.\n- The problem domain is intuitive, providing the potential to be expanded by others as additional points of interest arise. \n- Open challenges in ILP are well articulated, and the role of this dataset in providing a means to evaluate future developments in the field is conveyed clearly.", "weaknesses": "- The predicate arity section may not be ideally motivated.  For example, the (teacher, subject, student) relationship would probably be addressed in practice by making the subject a predicate, since a relatively small set of subjects at that level of granularity exist.  While the authors' observation is valid, it but might not be compelling to someone unfamiliar with the nuance.\n- In addition to combinatorial techniques referenced heavily in the writeup, there are a wide variety of published neuro symbolic techniques.  One of the few highlighted in this paper is by Evans and Grefenstette.  Evans et. all discuss raw data challenges [1], and his dissertation [2] has extensive discussion of challenges and approach. \n\n[1] Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence (IJCAI-22)\n[2] Kant's Cognitive Architecture", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NyIoDwYbbg", "reviewer": "Reviewer_Qcx3", "summary": "The paper  introduce GeoILP, a large-scale synthetic dataset of diverse ILP tasks involving numerous aspects of language bias.\nThe dataset consists of geometry problems modeled from levels as varied as textbook exercises, promoting the development of methods that can handle complex language biases, higher arity, and multi-task learning.", "questions": "How strong is the generalization capability of the neural-symbolic model trained on this dataset? Can it solve tasks beyond geometric problems?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper is clearly written.\n\n2. The dataset GEOILP is  large-scale and mimics real-world data better than traditional closed-world datasets.", "weaknesses": "The article provides a lot of background information, which results in insufficient coverage of its own work. \nFirstly, the experimental section is difficult to support the contributions of this paper. \nAlso, although Section 5 offers a detailed introduction to the content of its dataset, it does not effectively highlight the characteristics of its dataset in comparison to other datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2NW74PsYHR", "reviewer": "Reviewer_Prcn", "summary": "This paper presents a novel Inductive Logic Programming (ILP) dataset to evaluate ILP algorithms and systems. The presented GeoILP tasks try to formulate geometrical background knowledge as logic programs, and the motivation is to learn geometrical theories/concepts from them. Moreover, the authors also visualised all the tasks and produced images for the problems. The difficulty of the tasks is varied, ranging from simple questions to IMO-level questions.", "questions": "- Induction will be a very challenging problem for LLMs, would it be possible to extend the dataset and make a natural language version, so that people can compare ILP with LLMs?\n- Why not use formal languages that are designed specifically for automated proof, such as lean4 or Coq?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- The paper discussed the challenges for the current ILP area in detail, and the authors clearly understood the intrinsic disadvantages of ILP, thus the dataset is designed to motivate the community to resolve these long-neglected issues.\n- The authors have covered extensive related works, and the paper is well structured and well written.", "weaknesses": "- The design of the dataset is thoughtful, however, it is still like the previous ILP tasks, which are not very accessible to the ICLR community. For example, the representation of Logic Programming or Prolog is not user-friendly for normal users. The definition and theorems in plane geometry described with logic programs in this paper sometimes are difficult to understand.\n- Prolog is not a popular language for theorem proving, modern machine learning techniques such as LLMs could not handle them well enough.\n- The experiments in this paper are not enough, it is only experimented with Popper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xgZs2VYUE0", "reviewer": "Reviewer_swwC", "summary": "This paper proposed a new dataset in the geometry domain to help the researcher enhance the inductive logic programming (ILP) models. The authors indicated that there is no reference hypothesis in the current ILP datasets. This paper proposes a larger-scale ILP dataset with reference hypotheses. This paper also describes the algorithm for how to generate the synthesis data and the corresponding reference. \n\nHowever, the paper is borderline rejected for the following reasons: (1) The motivation of the paper is not explicitly discussed in the paper. (2) The Experiments are not explicitly investigated in the paper. Hence, the overall contribution of the paper is limited. (3) Some terminology is vague to use such as learning from raw data and open-world assumption.", "questions": "1. Why is this dataset helpful in solving the ILP problem? \n    1. Having reference hypotheses to guide the evaluation of the ILP model is not essential in all senses. In some ILP datasets proposed by [1] or FB15KSelected, the knowledge graph is easy to understand. Hence, there is no need to have a reference in addition. \n    2. Besides, some ILP models support precision and recall as quantitative metrics to evaluate the learned rules from data [3]. \n    3. Furthermore, when generating a set of rules, how to evaluate the performance of an ILP model based on the proposed reference hypotheses. Is the reference complete based on the background knowledge, positive examples, and negative examples?\n2. Based on these proposed datasets, the authors mentioned that no one ILP model can successfully learn rules from the GeoILP. The results are further explained in Section 6.2. The symbolic ILP models can not learn even one rule in the *basic level* setting. However, there is no explicit explanation about the basic level in line 520 page 10.  In addition, in line 524 on page 10, the authors stated that three neuro-symbolic models cannot solve the GeoILP because of the features of these ILP models. However, some neuro-symbolic ILP models can learn rules with three or more body atoms and any arities of a predicate [2]. The authors should also analyze more ILP models in Experiments to investigate the current performance of the ILP models.\n3. In addition, there is no reference in the Open-world assumption paragraph of Section  4.2. In addition, the open-world assumption is not clearly explained as to why the open-world assumption is related to intentional and extensional predicates in line 428 on page 8. The knowledge base uses the open-world assumption to determine the Boolean value of a ground atom, which is defined in line 257 on page 5. However, in line 428, the open-world assumption is applied based on the rule format. Hence, the authors should explain more about the connections between open-world assumptions and the format of rules. \n\nReference:\n[1] Richard Evans, Edward Grefenstette: Learning Explanatory Rules from Noisy Data. J. Artif. Intell. Res. 61: 1-64 (2018)\n[2] Xujie Si, Mukund Raghothaman, Kihong Heo, Mayur Naik: Synthesizing Datalog Programs using Numerical Relaxation. IJCAI 2019: 6117-6124\n[3] Tim Rocktäschel, Sebastian Riedel: End-to-end Differentiable Proving. NIPS 2017: 3788-3800", "limitations": "", "rating": 3, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper builds a novel ILP dataset to boost the development of the ILP community development. The proposed ILP is very challenging based on the statements of the paper. In addition, the authors present the methodology for generating the proposed ILP dataset.", "weaknesses": "1. Based on the structure of the paper, only Section 5 describes the proposed GeoILP. The rest of the Sections look like a survey to describe the development of the ILP methodology. Hence, the contribution of the paper including the method to generate the datasets and the evaluation of the proposed dataset for proving some properties such as learning recursive rules and long variables rules with the existing ILP models is still limited. \n2. When learning from raw data, the authors only discussed learning rules with the help of a pre-trained perception model. However, some discussions about learning from raw data directly without the symbolization process by the perception model are missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)