Title: Interpretable Graph Networks Formulate Universal Algebra Conjectures
Venue: NeurIPS 2023
Accepted: yes
Keywords: universal algebra

interpretability

graph neural networks

concept-based models
Abstract: The rise of Artificial Intelligence (AI) recently empowered researchers to investigate hard mathematical problems which eluded traditional approaches for decades. Yet, the use of AI in Universal Algebra (UA)---one of the fields laying the foundations of modern mathematics---is still completely unexplored. 
This work proposes the first use of AI to investigate UA's conjectures with an equivalent equational and topological characterization. While topological representations would enable the analysis of such properties using graph neural networks, the limited transparency and brittle explainability of these models hinder their straightforward use to empirically validate existing conjectures or to formulate new ones. 
To bridge these gaps, we propose a general algorithm generating AI-ready datasets based on UA's conjectures, and introduce a novel neural layer to build fully interpretable graph networks. The results of our experiments demonstrate that interpretable graph networks: (i) enhance interpretability without sacrificing task accuracy, (ii) strongly generalize when predicting universal algebra's properties, (iii) generate simple explanations that empirically validate existing conjectures, and (iv) identify subgraphs suggesting the formulation of novel conjectures.

Interpretable Graph Networks Formulate Universal Algebra Conjectures

Abstract

The rise of Artificial Intelligence (AI) recently empowered researchers to investigate hard mathematical problems which eluded traditional approaches for decades. Yet, the use of AI in Universal Algebra (UA)-one of the fields laying the foundations of modern mathematics-is still completely unexplored. This work proposes the first use of AI to investigate UA's conjectures with an equivalent equational and topological characterization. While topological representations would enable the analysis of such properties using graph neural networks, the limited transparency and brittle explainability of these models hinder their straightforward use to empirically validate existing conjectures or to formulate new ones. To bridge these gaps, we propose a general algorithm generating AI-ready datasets based on UA's conjectures, and introduce a novel neural layer to build fully interpretable graph networks. The results of our experiments demonstrate that interpretable graph networks: (i) enhance interpretability without sacrificing task accuracy, (ii) strongly generalize when predicting universal algebra's properties, (iii) generate simple explanations that empirically validate existing conjectures, and (iv) identify subgraphs suggesting the formulation of novel conjectures.

1 Introduction

Universal Algebra (UA, (6)) is one of the foundational fields of modern Mathematics with possible deep impact in all mathematical disciplines, but the complexity of studying abstract algebraic structures hinders scientific progress and discourages many academics. Recently, the emergence of powerful AI technologies empowered researchers to investigate intricate mathematical problems which eluded traditional approaches for decades, leading to the solution of open problems (e.g., (24)) and discovery of new conjectures (e.g., (7)). Yet, universal algebra currently remains an uninvestigated realm for AI, primarily for two reasons (i) first

FRAME

CONJECTURE

GENERATE

DATA SET

TRAIN AI

MODEL

EXTRACT

EXPLANATIONS

not modular

not modular

modular

iGNN

not modular

UA deals with infinite objects or even classes of abstract objects, which pose unique challenges for conventional AI techniques, (ii) secondly the field commonly relies on deterministic algorithms, utilized to construct finite models or serve as deterministic theorem provers, such as "mace4" and "prover9" 1 . In this sense, we hope that our paper could represent a guidance to further explore the study of UA's problems with AI, e.g. by integrating existing systems, like these mentioned theorem provers, with our methodology, which may suggest novel conjectures, in a unique scheme.

Universal algebra studies algebraic structures from an abstract perspective. Interestingly, several UA conjectures equivalently characterize algebraic properties using equations or graphs (16). In theory, studying UA properties as graphs would enable the use of powerful AI techniques, such as Graph Neural Networks (GNN, (39)), which excel on graph-structured data. However, two factors currently limit scientific progress. First, the absence of benchmark datasets suitable for machine learning prevents widespread application of AI to UA. Second, GNNs' opaque reasoning obstructs human understanding of their decision process (38). Compounding the issue of GNNs' limited transparency, GNN explainability methods mostly rely on brittle and untrustworthy local/post-hoc methods (14; 27; 28; 38; 45) or pre-defined subgraphs for explanations (2; 42), which are often unknown in UA.

Contributions. In this work, we investigate universal algebra's conjectures through AI (Figure 1). Our work includes three significant contributions. First, we propose a novel algorithm that generates a dataset suitable for training AI models based on an UA equational conjecture. Second, we generate and release the first-ever universal algebra's dataset compatible with AI, which contains more than 29 , 000 lattices and the labels of 5 key properties i.e., modularity, distributivity, semi-distributivity, join semi-distributivity, and meet semi-distributivity. And third, we introduce a novel neural layer that makes GNNs fully interpretable, according to Rudin's (38) notion of interpretability. The results of our experiments demonstrate that interpretable GNNs (iGNNs): (i) enhance GNN interpretability without sacrificing task accuracy, (ii) strongly generalize when trained to predict universal algebra's properties, (iii) generate simple concept-based explanations that empirically validate existing conjectures, and (iv) identify subgraphs which could be relevant for the formulation of novel conjectures. As a consequence, our findings demonstrate the potentiality of AI methods for investigating UA problems.

2 Background

Universal Algebra is a branch of mathematics studying general and abstract algebraic structures. Algebraic structures are typically represented as ordered pairs A = ( A,F ) , consisting of a non-empty set A and a collection of operations F defined on the set. UA aims to identify algebraic properties (often in equational form) shared by various mathematical systems. In particular, varieties are classes of algebraic structures sharing a common set of identities, which enable the study of algebraic systems based on their common properties. Prominent instances of varieties that have been extensively studied across various academic fields encompass Groups, Rings, Boolean Algebras, Fields, and many others. A particularly relevant variety of algebras are Lattices (details in Appendix A.3), which are often studied for their connection with logical structures.

Definition 2.1. A lattice L is an algebraic structure composed by a non-empty set L and two binary operations ∨ and ∧ , satisfying the commutativity, associativity, idempotency, and absorption axioms.

Equivalently a lattice can be characterized as a partially ordered set in which every pair of elements has a supremum ( ∨ ) and an infimum ( ∧ ) (cf. Appendix A.3). Lattices also have formal representations as graphs via Hasse diagrams ( L, E ) (e.g., Figure 2), where each node x ∈ L is a lattice element, and directed 2 edges ( x, y ) ∈ E ⊆ L × L represent the ordering relation, such that if ( x, y ) ∈ E then x ≤ L y in the ordering of the lattice. A sublattice L ′ of a lattice L is a lattice such that L ′ ⊆ L and

L ′ preserves the original order (the 'essential structure') of L , i.e. for all x, y ∈ L ′ then x ≤ L ′ y if and only if x ≤ L y . The foundational work by Birkhoff (5), Dedekind (9), and Jónsson (18) played a significant role in discovering that some significant varieties of lattices can be characterized through the omission of one or more lattices. Specifically, a variety V of lattices is said to omit a lattice L if the latter cannot be identified as a sublattice of any lattice in V . A parallel line of work in UA

characterizes lattices in terms of equational (" term 1 ≈ term 2 ") and quasi-equational ("if equation 1 holds then equation 2 holds") properties, such as distributivity and modularity.

Definition 2.2. Let L be a lattice. L is modular if it satisfies x ≤ y → x ∨ ( y ∧ z ) ≈ y ∧ ( x ∨ z ) ; distributive if it satisfies x ∨ ( y ∧ z ) ≈ ( x ∨ y ) ∧ ( x ∨ z ) .

We notice that each distributive lattice is also modular (see Appendix A.3). Figure 2 represents M 3 (modular with 3 atoms) and N 5 (non-modular with 5 atoms), the smallest instances of lattices which exhibit failures of distributivity and modularity, respectively. For example, N 5 is neither modular nor distributive, considering the substitution x = a, y = c, z = b . The same substitution shows that M 3 is not distributive. The classes of distributive and modular lattices show classical examples of varieties that can equivalently be characterized using equations and lattice omissions, as illustrated by the following theorems.

Theorem 2.3 (Dedekind (9)) . A lattice variety V is modular if and only if V omits N 5 .

Theorem 2.4 (Birkhoff (5)) . A lattice variety V is distributive if and only if V omits N 5 and M 3 .

Starting from these classic results, the investigation of lattice omissions and the structural characterizations of classes of lattices has evolved into a rich and extensively studied field (16), but it was never approached with AI methodologies before.

3 Methods

The problem of characterizing lattice varieties through lattice omission is very challenging as it requires the analysis of large (potentially infinite) lattices (5; 9; 18). To address this task, we propose the first AI-assisted framework supporting mathematicians in finding empirical evidences to validate existing conjectures and to suggest novel theorems. To this end, we propose a general algorithm (Section 3.1) allowing researchers in universal algebra to define a property of interest and generate a dataset suitable to train AI models. We then introduce interpretable graph networks (Section 3.2) which can suggest candidate lattices whose omission is responsible for the satisfaction of the given algebraic property.

3.1 A Tool to Generate Datasets of Lattice Varieties

We propose a general methodology to investigate any algebraic property whose validity can be verified on a finite lattice. In this work, we focus on properties that can be characterized via equations and quasi-equations. To train AI models, we propose a general dataset generator 3 for lattice varieties. Intuitively, the generator takes as input the number of nodes n in the lattices and a function to check whether a lattice satisfies a given property. We generate 2 n × n matrices of size n × n , containing all binary functions

Algorithm 1:

Generate dataset of lattice varieties.

Input:

n

1

,

hasProperty

(

·

,

·

,

·

)

//

n

: cardinality

Dataset

= []

AllFuncs

genAllFuncs

(

n

)

//

binary functions as

n

×

n

matrices

for

L

∈

AllFuncs

do

//

L

(

i, j

) = 1

meaning

i

L

j

if

isPartialOrder

(

L

)

then

//

check if

L

is refl., antisym. and trans.

if

isLattice

(

L

)

then

//

check if

L

is a lattice

for

i, j

n

do

∧

L

[

i, j

]

sup

x

L

{

x

L

i

and

x

L

j

}

∨

L

[

i, j

]

inf

x

L

{

i

L

x

and

j

L

x

}

if

hasProperty

(

L,

∧

L

,

∨

L

)

then

//

check

∧

L

,

∨

L

properties

Dataset

.append

([

L,

True

])

else

Dataset

.append

([

L,

False

])

definable on { 1 , . . . , n } 2 , and filter only binary matrices representing partial orders. Then, we verify that the partial ordered set L is a lattice, by checking that any pair of nodes always has a unique infimum and supremum. This directly verifies that ∧ L and ∨ L satisfy Definition 2.1. Finally, we check whether the lattice satisfies the target property or not, and append it and the property label to our dataset. We remark that checking the validity of a single ternary equation on a medium-size lattice is not computationally prohibitive (i.e., it 'only' requires checking n 3 identities), but the number of existing lattices increases exponentially as n increases. For instance, it is known that there are at least 2,000,000 non-isomorphic lattices with n = 10 elements (4). Therefore, we only sample a fixed number of lattices per cardinality starting from a certain node cardinality. While this may seem a strong bias, we notice that known and relevant lattice omissions often rely on lattices with few nodes (5; 9). To empirically verify that this is not a significant limitation, in our experiments we deliberately investigate the generalization capacity of GNNs when trained on small-size lattices and tested on larger ones. This way we can use GNNs to predict the satisfiability of equational properties

MESSAGE PASSING

HARD

GUMBEL-SOFTMAX

INTERPRETABLE

CLASSIFIER

CONCEPT

"2-node neighborhood"

CONCEPT

"3-node neighborhood"

INTERPRETABLE

AGGREGATION

(SUM/MAX)

(LINEAR MODEL)

1

0

0

0

0

1

"NON DISTRIBUTIVE"

on large graph structures without explicitly checking them. Using Algorithm 1, we generated the first large-scale AI-compatible datasets of lattices containing more than 29 , 000 graphs and the labels of 5 key properties of lattice (quasi-)varieties i.e., modularity, distributivity, semi-distributivity, join semi-distributivity, and meet semi-distributivity, whose definitions can be found in Appendix A.

Scalability and Complexity. The scalability of the dataset generation process is related to two main points: (i) dealing with the exponential growth of the number of lattices, (ii) how to feasibly generate a lattice of a finite amount of elements. (i) Since the number of lattices definable on a set L = { 0 , ...n -1 } increases exponentially with n , it is not feasible to realize a dataset containing all the lattices of n elements. However, this is also unnecessary for the scope of this work, which aims at automatically identifying (small) topological patterns responsible for the failure of certain algebraic properties. In addition, the majority of the graphs generated are isomorphic, thus not particularly informative for our task. Because of this, we only generate a portion of all the lattices up to a certain value M > 0 , and then take a fixed number of samples n s up to N (we take M = 8 , n s = 20 , N = 50 in the paper). (ii) The construction of a lattice with n elements is briefly sketched in Algorithm 1. We refer to Appendix B for the technical details on the design of the used functions.

3.2 Interpretable Graph Networks (iGNNs)

In this section, we design an interpretable graph network (iGNN, Figure 3) that satisfies the notion of "interpretability" introduced by Rudin (38). According to this definition, a machine learning (ML) system is interpretable if and only if (1) its inputs are semantically meaningful, and (2) its model inference is simple for humans to understand (e.g., sparse and/or symbolic). This definition covers ML systems that take tabular datasets or sets of concepts as inputs, and (piece-wise) linear models such as logistic regression or decision trees. To achieve this goal in GNNs, we introduce an interpretable graph layer that learns semantically meaningful concepts and uses them as inputs for a simple linear classification layer. We then show how this layer can be included into existing architectures or into hierarchical iGNNs, which consist of a sequence of interpretable graph layers. We notice that in devising our approach, we preferred to rely on linear classifiers as they are fully differentiable, hence allowing us to realize a fully interpretable and differentiable model from the input to the classification head. However,in practice any interpretable and differentiable classifier could be used in place of this linear layer.

3.2.1 Interpretable Graph Layer

The interpretable graph layer (Figure 3) serves three main functions: message passing, concept generation, and task predictions. The first step of the interpretable graph layer involves a standard message passing operation (Eq. 1 right), which aggregates information from node neighbors. This operation enables to share and process relational information across nodes and it represents the basis of any GNN layer.

Node-level concepts. An interpretable concept space is the first step towards interpretability. Following Ghorbani et al. (10), a relevant concept is a 'high-level human-understandable unit of

information' shared by input samples and thus identifiable with clustering techniques. Message passing algorithms do cluster node embeddings based on the structure of node neighborhoods, as observed by Magister et al. (28). However, the real-valued large embedding representations h i ∈ R q , q ∈ N generated by message passing can be challenging for humans to interpret. To address this, we use a hard Gumbel-Softmax activation Θ : R q ↦→{ 0 , 1 } q , following Azzolin et al. (2):

where ψ is a learnable function ignoring or assuming constant input features and ϕ is a learnable function aggregating information from a node neighborhood N i , and ⊕ is a permutation invariant aggregation function (such as sum or mean). During the forward pass, the Gumbel-Softmax activation Θ produces a one-hot encoded representation of each node embedding. Since nodes sharing the same neighborhood have similar embeddings h i due to message passing, they will also have the same one-hot vector c i due to the Gumbel-Softmax, and vice versa - we can then interpret nodes having the same one-hot concept c i as nodes having similar embeddings h i and thus sharing a similar neighborhood. More formally, we can assign a semantic meaning to a reference concept γ ∈ { 0 , 1 } q by visualizing concept prototypes corresponding to the inverse images of a node concept vector. In practice, we can consider a subset of the input lattices Γ corresponding to the node's ( p -hop) neighborhood covered by message passing:

where D is the set of all training lattices, and L ⟨ i,p ⟩ is the graph corresponding to the p -hop neighborhood ( p ∈ { 1 , . . . , | L |} ) of the node i ∈ L , as suggested by Ghorbani et al. (10); Magister et al. (28). This way, by visualizing concept prototypes as subgraph neighborhoods, the meaning of the concept representation becomes easily interpretable to humans (Figure 3), aiding in the understanding of the reasoning process of the network.

Example 3.1 (Interpreting node-level concepts) . Consider the problem of classifying distributive lattices with a simplified dataset including N 5 and M 3 only, and where each node has a constant feature x i = 1 . As these two lattices only have nodes with 2 or 3 neighbours, one layer of message passing will then generate only two types of node embeddings e.g., h II = [0 . 2 , -0 . 4 , 0 . 3] for nodes with a 2-nodes neighborhood (e.g., ), and h III = [0 . 6 , 0 . 2 , -0 . 1] for nodes with a 3-nodes neighborhood (e.g., ). As a consequence, the Gumbel-Softmax will only generate two possible concept vectors e.g., c II = [0 , 0 , 1] and c III = [1 , 0 , 0] . Hence, for instance the concept belongs to c II , while belongs to c III .

Graph-level concept embeddings. To generate a graph-level concept space in the interpretable graph layer, we can utilize the node-level concept space produced by the Gumbel-Softmax. Normally, graph-level embeddings are generated by applying a permutation invariant aggregation function on node embeddings. However, in iGNNs we restrict the options to (piece-wise) linear permutation invariant functions in order to follow our interpretability requirements dictated by Rudin (38). This restriction still includes common options such as max or sum pooling. Max pooling can easily be interpreted by taking the component-wise max over the one-hot encoded concept vectors c i . After max pooling, the graph-level concept vector has a value of 1 at the k -th index if and only if at least one node activates the k -th concept i.e., ∃ i ∈ L, c ik = 1 . Similarly, we can interpret the output of a sum pooling: a graph-level concept vector takes a value v ∈ N at the k -th index after sum pooling if and only if there are exactly v nodes activating the k -th concept i.e., ∃ i 0 , . . . , i v ∈ L, c ik = 1 .

Example 3.2 (Interpreting graph-level concepts) . Following Example 3.1, let us use sum pooling to generate graph-level concepts. For an N 5 graph, we have 5 nodes with exactly the same 2-node neighborhood. Therefore, sum pooling generates a graph-level embedding [0 , 0 , 5] , which certifies that we have 5 nodes of the same type e.g., . For an M 3 graph, the top and bottom nodes have a 3-node neighborhood e.g., , while the middle nodes have a 2-node neighborhood e.g., . This means that sum pooling generates a graph-level embedding [2 , 0 , 3] , certifying that we have 2 nodes of type and 3 nodes of type .

Interpretable classifier. To prioritize the identification of relevant concepts, we use a classifier to predict the task labels using the concept representations. A black-box classifier like a multi-layer perceptron (36) would not be ideal as it could compromise the interpretability of our model, so instead we use an interpretable linear classifier such as a single-layer network (20). This allows for a completely interpretable and differentiable model from the input to the classification head, as the input representations of the classifier are interpretable concepts and the classifier is a simple linear model which is intrinsically interpretable as discussed by Rudin (38). In fact, the weights of the perceptron can be used to identify which concepts are most relevant for the classification task. Hence, the resulting model can be used not only for classification, but also to interpret and understand the problem at hand.

3.2.2 Interpretable architectures

The interpretable graph layer can be used to instantiate different types of iGNNs. One approach is to plug this layer as the last message passing layer of a standard GNN architecture:

where f is an interpretable classifier (e.g., single-layer network), ⊞ is an interpretable piece-wise linear and permutation-invariant function (such as max or sum), Θ is a Gumbel-Softmax hard activation function, and h 0 i = x i . In this way, we can interpret the first part of the network as a feature extractor generating well-clustered latent representations from which concepts can be extracted. This approach is useful when we only care about the most complex neighborhoods/concepts. Another approach is to generate a hierarchical transparent architecture where each GNN layer is interpretable:

In this case, we can interpret every single layer of our model with concepts of increasing complexity. The concepts extracted from the first layer represent subgraphs corresponding to the 1 -hop neighborhood of a node, those extracted at the second layer will correspond to 2 -hop neighborhoods, and so on. These hierarchical iGNNs can be useful to get insights into concepts with different granularities. By analyzing the concepts extracted at each layer, we gain a better understanding of the GNN inference and of the importance of different (sub)graph structures for the classification task.

3.2.3 Training

For the classification layer, the choice of the activation and loss functions for iGNNs depends on the nature of the task at hand and does not affect their interpretability. For classification tasks, we use standard activation functions such as softmax or sigmoid, along with standard loss functions like cross-entropy. For hierarchical iGNNs (HiGNNs), we apply the same loss function at each layer of the concept hierarchy, as their layered architecture enables intermediate supervisions. This ensures that each layer is doing its best to extract the most relevant concepts to solve the task. Internal losses can also be weighted differently to prioritize the formation of optimal concepts of a specific size, allowing the HiGNN to learn in a progressive and efficient way.

4 Experimental Analysis

4.1 Research questions

In this section we analyze the following research questions:

Generalization - Can GNNs generalize when trained to predict universal algebra's properties? Can interpretable GNNs generalize as well?

Interpretability - Do interpretable GNNs concepts empirically validate universal algebra's conjectures? How can concept-based explanations suggest novel conjectures?

4.2 Setup

Baselines. For our comparative study, we evaluate the performance of iGNNs and their hierarchical version against equivalent GNN models (i.e., having the same hyperparameters such as number layers, training epochs, and learning rate). For vanilla GNNs we resort to common practice replacing the Gumbel-Softmax with a standard leaky ReLU activation. We exclude from our main baselines prototype or concept-based GNNs pre-defining graph structures for explanations, as for most datasets these structures are unknown. Appendix C covers implementation details. We show more extensive results including local and post-hoc explanations in Appendix G.

Evaluation. We employ three quantitative metrics to assess a model's generalization and interpretability. We use the Area Under the Receiver Operating Characteristic (AUC ROC) curve to assess task generalization. We evaluate generalization under two different conditions: with independently and identically distributed train/test splits, and out-of-distribution by training on graphs up to eight nodes, while testing on graphs with more than eight nodes ('strong generalization' (41)). We further assess generalization under binary and multilabel settings (classifying 5 properties of a lattice at the same time). To evaluate interpretability, we use standard metrics such as completeness (44) and fidelity (37). Completeness 4 assesses the quality of the concept space on a global scale using an interpretable model to map concepts to tasks, while fidelity measures the difference in predictions obtained with an interpretable surrogate model and the original model. Finally, we evaluate the meaningfulness of our concept-based explanations by visualizing and comparing the generated concepts with ground truth lattices like e.g. M 3 and N 5 , whose omission is known to be significant for modular and distributive properties. All metrics in our evaluation, across all experiments, are computed on test sets using 5 random seeds, and reported using the mean and 95% confidence interval.

5 Key Findings

5.1 Generalization

iGNNs improve interpretability without sacrificing task accuracy (Figure 4). Our experimental evaluation reveals that interpretable GNNs are able to strike a balance between completeness and fidelity, two crucial metrics that are used to assess generalization-interpretability trade-offs (37). We observe that the multilabel classification scenario, which requires models to learn a more varied and diverse set of concepts, is the most challenging and results in the lowest completeness scores on average. We also notice that the more challenging out-of-distribution scenario results in the lowest completeness and fidelity scores across all datasets. More importantly, our findings indicate that iGNNs achieve optimal fidelity scores, as their classification layer consists of a simple linear function of the learnt concepts which is intrinsically interpretable (38). On the contrary, interpretable surrogate models of black-box GNNs exhibit, as expected, lower fidelity scores, confirming analogous observations in the explainable AI literature (37; 38). In practice, this discrepancy between the original black-box predictions and the predictions obtained with an interpretable surrogate model questions the actual usefulness of black-boxes when interpretable alternatives achieve similar results in solving the problem at hand, as extensively discussed by Rudin (38). Overall, these results demonstrate how concept spaces are highly informative to solve universal algebra's tasks and how the interpretable graph layer may improve GNNs' interpretability without sacrificing task accuracy. We refer the reader to Appendix E for detailed discussion on quantitative analysis of concept space obtained by iGNNs under different generalization settings with comparisons to their black-box counterparts.

GNNs strongly generalize on universal algebra's tasks (Figure 5). Our experimental findings demonstrate the strong generalization capabilities of GNNs across the universal algebra tasks we designed. Indeed, we stress GNNs test generalization abilities by training the models on graphs of size up to n (with n ranging from 5 to 8 ), and evaluating their performance on much larger graphs of size up to 50 . We designed this challenging experiment in order to understand the limits and robustness of interpretable GNNs when facing a significant data distribution shift from training to test data. Remarkably, iGNNs exhibit robust generalization abilities (similar to their black-box counterparts) when trained on graphs up to size 8 and tested on larger graphs. This evidence confirms the hypothesis that interpretable models can deliver reliable and interpretable predictions, as suggested by Rudin

80

90

100

COMPLETENESS %

80

85

90

95

100

FIDELITY %

multilabel

60

80

100

COMPLETENESS %

20

40

60

80

100

Distributive

60

80

100

COMPLETENESS %

40

60

80

100

Modular

80

90

100

COMPLETENESS %

70

80

90

100

MeetSemiDistributive

80

90

100

COMPLETENESS %

85

90

95

100

JoinSemiDistributive

85

90

95

100

COMPLETENESS %

85

90

95

100

SemiDistributive

GSAT

GSAT (strong)

Black-Box + GCExplainer

Black-Box + GCExplainer (strong)

HiGNN

HiGNN (strong)

iGNN

iGNN (strong)

(38). However, we observe that black-box GNNs slightly outperform iGNNs when trained on even smaller lattices. We hypothesize that this is due to the more constrained architecture of iGNNs, which imposes tighter bounds on their expressiveness when compared to standard black-box GNNs.

Notably, training with graphs of size up to 5 or 6 significantly diminishes GNNs generalization in the tasks we designed. We hypothesize that this is due to the scarcity of non-distributive and non-modular lattices during training, but it may also suggest that some patterns of size 7 and 8 might be quite relevant to generalize

5

6

7

8

50

60

70

80

90

100

Distributive

5

6

7

8

50

60

70

80

90

100

Modular

5

6

7

8

50

60

70

80

90

100

Multilabel

Maximum Number of Nodes in Training

Test AUCROC (%)

Black-Box GNN

iGNN

HiGNN

to larger graphs. Unfortunately, running generalization experiments with n ≤ 4 was not possible since all such lattices trivially omitted N 5 and M 3 . It is worth mentioning that GNNs performed well even in the challenging multilabel case, where they had to learn a wider and more diverse set of concepts and tasks. In all experiments, we observe a plateau of the AUC ROC scores for n = 8 , thus suggesting that a training set including graphs of this size might be sufficient to learn the relevant patterns allowing the generalization to larger lattice varieties. For detailed numerical results across all tasks, we refer the reader to Table 1 in Appendix D. Overall, these results emphasize the potential of GNNs in addressing complex problems in universal algebra, providing an effective tool to handle lattices that are difficult to analyze manually with pen and paper.

5.2 Interpretability

Concept-based explanations empirically validate universal algebra's conjectures (Figure 6). We present empirical evidence to support the validity of theorems 2.3 and 2.4 by examining the concepts generated for modular and distributive tasks. For this investigation we leverage the interpretable structure of iGNNs.

Similarly to Ribeiro et al. (37), we visualize in Figure 6 the weights of our trained linear classifier representing the relevance of each concept. We remark that the visualization is limited to the (top5 ) most negative weights, as we are interested in those concepts that negatively affect the prediction of a property. In the same plot, we also show the prototype of each concept represented by the 2-hop neighbor-

4

15

7

6

8

Concept ID [Distributive]

-

0

.

4

-

0

.

3

-

0

.

2

-

0

.

1

0

.

0

Omission Relevance

15

3

10

9

6

Concept ID [Modular]

-

0

.

3

-

0

.

2

-

0

.

1

0

.

0

Omission Relevance

hood of a node activating the concept, following a similar procedure as Ghorbani et al. (10); Magister et al. (28); Azzolin et al. (2). Using this visualization, we investigate the presence of certain concepts when classifying modular and distributive lattices. For the modularity task, our results show that the lattice N 5 appears among non-modular concepts, but is never found in modular lattices, while the lattice M 3 appears among both modular and non-modular concepts, which is consistent with Theorem 2.3. In the case of distributivity, we observe that both M 3 and N 5 are present among non-distributive concepts, and are never found in distributive lattices, which is also in line with Theorem 2.4. These findings provide a large-scale empirical evidence for the validity of theorems 2.3 and 2.4, and further demonstrate the effectiveness of graph neural networks in learning and analyzing lattice properties. Overall, these results highlight how interpretable GNNs can not only learn the properties of universal algebra but also identify structures that are unique to one type of lattice (e.g., non-modular) and absent from another (e.g., modular), thus providing human-interpretable explanations for what the models learn.

Contrastive explanations highlight topological differences between properties of lattice varieties (Figure 7). We leverage interpretable GNNs to analyze the key topological differences of classical lattice properties such as join and meet semi-distribuitivity characterized by relevant quasi-equations (cf. Appendix A.5). To this end, we visualize specific concept prototypes corresponding to lattices that are not meet semi-distributive against lattices that are meet semi-distributive.

We observe N 5 but not M 3 among the concepts of meet semi-distributive lattices, while we observe both N 5 and M 3 only in concepts that are not meet semi-distributive. This observation suggests that N 5 is not a key lattice for meet semi-distributive

Non Join Semi-Distributive Lattice Varieties Non Meet Semi-Distributive Lattice Varieties

Meet Semi-Distributive

Meet Semi-Distributive

Meet Semi-Distributive

Join Semi-Distributive

Join Semi-Distributive

Join Semi-Distributive

lattices, unlike distributive lattices . Furthermore, we find that the lattice pattern is relevant for non meet semidistributivity, while its dual is relevant for non join semidistributivity, thus empirically confirming the hypotheses of Jónsson and Rival (19). These findings are significant because they demonstrate how analyzing concepts in interpretable GNNs can provide universal algebraists with a powerful and automatic tool to formulate new conjectures based on identifying simple lattices that play a role in specific properties. By leveraging the power of interpretable GNNs, we may uncover previously unknown connections between different properties and identify new patterns and structures that could lead to the development of new conjectures and theorems in universal algebra, providing exciting opportunities for future research in universal algebra.

6 Discussion

Relations with Graph Neural Network explainability. Graph Neural Networks (GNNs,(39)) process relational data generating node representations by combining the information of a node with that of its neighbors, thanks to a general learning paradigm known as message passing (11). A number of post-hoc explainability techniques have been proposed to explain the reasoning of GNNs. Inspired by vision approaches (40; 37; 10), early explainability techniques focused on feature importance (35), while subsequent works aimed to extract local explanations (45; 27; 42) or global explanations using conceptual subgraphs by clustering the activation space (28; 46; 29; 22). However, all these techniques either rely on pre-defined subgraphs for explanations (which are often unknown in UA) or provide post-hoc explanations which may be brittle and unfaithful as extensively demonstrated by Rudin (38). On the contrary, our experiments show that iGNNs generate interpretable predictions according to Rudin (38) notion of interpretability via linear classifiers applied on sparse human-understandable concept representations.

Limitations. The approach proposed in this paper focuses on universal algebra conjectures characterized both algebraically and topologically. Our methodology is limited to finite lattices, which may not capture all relevant information about infinite algebraic structures. However, the insights gained from finite-lattice explanations can still provide valuable information regarding a given problem (albeit with potentially limited generalization). Moreover, our approach is restricted to topological

properties on graphs, while non-structural properties may require the adoption of other kinds of (interpretable) models.

Broader impact and perspectives. AI techniques are becoming increasingly popular for solving previously intractable mathematical problems and proposing new conjectures (23; 26; 7; 12). However, the use of modern AI methods in universal algebra was a novel and unexplored field until the development of the approach presented in this paper. To this end, our method uses interpretable graph networks to suggest graph structures that characterize relevant algebraic properties of lattices. With our approach, we empirically validated Dedekind (9) and Birkhoff (5) theorems on distributive and modular lattices, by recovering relevant lattices. This approach can be readily extended-beyond equational properties determined by the omission of a sublattice in a variety (43)to any structural property of lattices, including the characterization of congruence lattices of algebraic varieties (1; 21; 31; 43). Our methodology can also be applied (beyond universal algebra) to investigate (almost) any mathematical property that can be topologically characterized on a graph, such as the classes of graphs/diagraphs with a fixed set of polymorphisms (25; 3; 32).

Conclusion. This paper presents the first-ever AI-assisted approach to investigate equational and topological conjectures in the field of universal algebra. To this end, we present a novel algorithm to generate datasets suitable for AI models to study equational properties of lattice varieties. While topological representations would enable the use of graph neural networks, the limited transparency and brittle explainability of these models hinder their use in validating existing conjectures or proposing new ones. For this reason, we introduce a novel neural layer to build fully interpretable graph networks to analyze the generated datasets. The results of our experiments demonstrate that interpretable graph networks: enhance interpretability without sacrificing task accuracy, strongly generalize when predicting universal algebra's properties, generate simple explanations that empirically validate existing conjectures, and identify subgraphs suggesting the formulation of novel conjectures. These promising results demonstrate the potential of our methodology, opening the doors of universal algebra to AI with far-reaching impact across all mathematical disciplines.

Acknowledgments and Disclosure of Funding

This paper was supported by TAILOR and by HumanE-AI-Net projects funded by EU Horizon 2020 research and innovation programme under GA No 952215 and No 952026, respectively. This paper has been also supported by the Austrian Science Fund FWF project P33878 'Equations in Universal Algebra' and the European Union's Horizon 2020 research and innovation programme under grant agreement No 848077. This project has also received funding from the European Union's Horizon-MSCA-2021 research and innovation program under grant agreement No 101073307.

References

Paolo Aglianò, Stefano Bartali, and Stefano Fioravanti. On Freese's technique. International Journal of Algebra and Computation , 2023. doi: https://doi.org/10.1142/S0218196723500601. URL https://arxiv.org/abs/2302.11452 .

Steve Azzolin, Antonio Longa, Pietro Barbiero, Pietro Liò, and Andrea Passerini. Global explainability of gnns via logic combination of learned concepts. arXiv preprint arXiv:2210.07147 , 2022.

Libor Barto, Marcin Kozik, and Todd Niven. The csp dichotomy holds for digraphs with no sources and no sinks (a positive answer to a conjecture of bang-jensen and hell). SIAM Journal on Computing , 38(5):1782-1802, 2009.

Joel Berman and Paweł Idziak. Counting finite algebras in the post varieties. International Journal of Algebra and Computation , 10(03):323-337, 2000.

Garrett Birkhoff. On the structure of abstract algebras. In Mathematical proceedings of the Cambridge philosophical society , volume 31, pages 433-454. Cambridge University Press, 1935.

Stanley Burris and H. P. Sankappanavar. A Course in Universal Algebra . Springer, 1981.

Alex Davies, Petar Veliˇ ckovi´ c, Lars Buesing, Sam Blackwell, Daniel Zheng, Nenad Tomašev, Richard Tanburn, Peter Battaglia, Charles Blundell, András Juhász, Marc Lackenby, Geordie Williamson, Demis Hassabis, and Pushmeet Kohli. Advancing mathematics by guiding human intuition with AI. Nature , 600(7887):70-74, December 2021. doi: 10.1038/ s41586-021-04086-x. URL https://doi.org/10.1038/s41586-021-04086-x .

Alan Day. A characterization of modularity for congruence lattices of algebras*. Canadian Mathematical Bulletin , 12(2):167-173, 1969. doi: 10.4153/CMB-1969-016-6.

Richard Dedekind. Über die von drei Moduln erzeugte Dualgruppe. Math. Ann. , 1900.

Amirata Ghorbani, James Wexler, James Y Zou, and Been Kim. Towards automatic conceptbased explanations. Advances in Neural Information Processing Systems , 32, 2019.

Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine learning , pages 1263-1272. PMLR, 2017.

Yang-Hui He. Machine-learning mathematical structures. International Journal of Data Science in the Mathematical Sciences , pages 1-25, 2022.

Jobst Heitzig and Jürgen Reinhold. Counting finite lattices. Algebra universalis , 48(1):43-53, 2002.

Qiang Huang, Makoto Yamada, Yuan Tian, Dinesh Singh, and Yi Chang. Graphlime: Local interpretable model explanations for graph neural networks. IEEE Transactions on Knowledge and Data Engineering , 2022.

Martin Hyland and John Power. The category theoretic understanding of universal algebra: Lawvere theories and monads. Electronic Notes in Theoretical Computer Science , 172:437-458, 2007. ISSN 1571-0661. doi: https://doi.org/10.1016/j.entcs.2007.02.019. URL https://www.sciencedirect.com/science/article/pii/S1571066107000874 . Computation, Meaning, and Logic: Articles dedicated to Gordon Plotkin.

Peter Jipsen and Henry Rose. Varieties of Lattices . Springer Berlin, 1992.

Bjarni Jonnson. Algebras whose congruence lattices are distributive. MATHEMATICA SCANDINAVICA , 21:110-121, Dec. 1967. doi: 10.7146/math.scand.a-10850. URL https: //www.mscand.dk/article/view/10850 .

Bjarni Jónsson. On the representation of lattices. Mathematica Scandinavica , 1953.

Bjarni Jónsson and Ivan Rival. Lattice varieties covering the smallest non-modular lattice variety. Pacific J. Math. , Volume 82, 1978.

Samandarov Erkaboy Karimboyevich and Abdurakhmonov Olim Nematullayevich. Single layer artificial neural network: Perceptron. European Multidisciplinary Journal of Modern Science , 5:230-238, Apr. 2022. URL https://emjms.academicjournal.io/index.php/emjms/ article/view/253 .

Keith Kearnes and Emil Kiss. The shape of congruence lattices. Memoirs of the American Math. Soc. , 2013.

Oguzhan Keskin, Alisia Maria Lupidi, Stefano Fioravanti, Lucie Charlotte Magister, Pietro Barbiero, Pietro Lio, and Francesco Giannini. Bridging equational properties and patterns on graphs: an ai-based approach. In Topological, Algebraic and Geometric Learning Workshops 2023 , pages 156-168. PMLR, 2023.

Guillaume Lample and François Charton. Deep learning for symbolic mathematics. In International Conference on Learning Representations .

Guillaume Lample and François Charton. Deep learning for symbolic mathematics. arXiv preprint arXiv:1912.01412 , 2019.

Barto Libor and Kozik Marcin. Absorbing Subalgebras, Cyclic Terms, and the Constraint Satisfaction Problem. Logical Methods in Computer Science , Volume 8, Issue 1, 2012.

Donald W Loveland. Automated theorem proving: A logical basis . Elsevier, 2016.

Dongsheng Luo, Wei Cheng, Dongkuan Xu, Wenchao Yu, Bo Zong, Haifeng Chen, and Xiang Zhang. Parameterized explainer for graph neural network. Advances in neural information processing systems , 33:19620-19631, 2020.

Lucie Charlotte Magister, Dmitry Kazhdan, Vikash Singh, and Pietro Liò. Gcexplainer: Human-in-the-loop concept-based explanations for graph neural networks. arXiv preprint arXiv:2107.11889 , 2021.

Lucie Charlotte Magister, Pietro Barbiero, Dmitry Kazhdan, Federico Siciliano, Gabriele Ciravegna, Fabrizio Silvestri, Mateja Jamnik, and Pietro Lio. Encoding concepts in graph neural networks. arXiv preprint arXiv:2207.13586 , 2022.

Siqi Miao, Mia Liu, and Pan Li. Interpretable and generalizable graph learning via stochastic attention mechanism. In International Conference on Machine Learning , pages 15524-15543. PMLR, 2022.

J. B. Nation. Varieties whose congruences satisfy certain lattice identities. Algebra Universalis , 1974.

Miroslav Olšák. The local loop lemma. Algebra universalis , 2020.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. arXiv preprint arXiv:1912.01703 , 2019.

Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikitlearn: Machine learning in python. the Journal of machine Learning research , 12:2825-2830, 2011.

Phillip E Pope, Soheil Kolouri, Mohammad Rostami, Charles E Martin, and Heiko Hoffmann. Explainability methods for graph convolutional neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10772-10781, 2019.

Marius-Constantin Popescu, Valentina Balas, Liliana Perescu-Popescu, and Nikos Mastorakis. Multilayer perceptron and neural networks. WSEAS Transactions on Circuits and Systems , 8, 07 2009.

Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. " why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining , pages 1135-1144, 2016.

Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature machine intelligence , 1(5):206-215, 2019.

Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE transactions on neural networks , 20(1):61-80, 2008.

Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034 , 2013.

Petar Veliˇ ckovi´ c, Rex Ying, Matilde Padovano, Raia Hadsell, and Charles Blundell. Neural execution of graph algorithms. In International Conference on Learning Representations , 2019.

Minh Vu and My T Thai. Pgm-explainer: Probabilistic graphical model explanations for graph neural networks. Advances in neural information processing systems , 33:12225-12235, 2020.

Philip M Whitman. Free lattices. Annals of Mathematics , pages 325-330, 1941.

Chih-Kuan Yeh, Been Kim, Sercan Arik, Chun-Liang Li, Tomas Pfister, and Pradeep Ravikumar. On completeness-aware concept-based explanations in deep neural networks. Advances in Neural Information Processing Systems , 33:20554-20565, 2020.

Rex Ying, Dylan Bourgeois, Jiaxuan You, Marinka Zitnik, and Jure Leskovec. Gnnexplainer: Generating explanations for graph neural networks, 2019.

Zaixi Zhang, Qi Liu, Hao Wang, Chengqiang Lu, and Cheekong Lee. Protgnn: Towards self-explaining graph neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 9127-9135, 2022.

A Algebra definitions

A.1 Formal defintions for Universal Algebra

Universal algebra is the field of mathematics that studies algebraic structures, which are defined as a set A along with its own collection of operations. In this section, we recall some basic definitions and theorems from Burris and Sankappanavar (6); Day (8); Jonnson (17), about elements of universal algebra and lattice theory.

Definition A.1. N-ary function For a non-empty set A and n non-negative integer we define A 0 = {∅} and, for n > 0 , A n is the set of n-tuples of elements from A . An n -ary operation on A is any function f from A n to A ; n is the arity of f . An operation f on A is called an n -ary operation if its arity is n .

Definition A.2. Algebraic Structure An algebra A is a pair ( A,F ) where A is a non-empty set called universe and F is a set of finitary operations on A .

Apart from the operations on A , an algebra is further defined by axioms, that in the particular case of universal algebras are in the form of identities.

Definition A.3. A lattice L is an algebraic structure composed by a non-empty set L and two binary operations ∨ and ∧ satisfying the following axioms and their duals obtained exchanging ∨ and ∧ :

x ≈ x ∨ ( x ∧ y )

x ∨ y ≈ y ∨ x

(commutativity)

x

∨

(

y

∨

z

)

≈

(

x

∨

y

)

(associativity)

x ∨ x ≈ x

(idempotency)

(absorption)

Theorem A.4 ((6)) . A partially ordered set L is a lattice if and only if for every a, b ∈ L both supremum and infimum of { a, b } exist (in L ) with a ∨ b being the supremum and a ∧ b the infimum.

Definition A.5. Let L be a lattice. Then L is modular ( distributive , ∨ -semi-distributive , ∧ -semidistributive ) if it satisfies the following:

x ≤ y → x ∨ ( y ∧ z ) ≈ y ∧ ( x ∨ z )

(modularity)

x ∨ ( y ∧ z ) ≈ ( x ∨ y ) ∧ ( x ∨ z )

(distributivity)

x ∨ y ≈ x ∨ z → x ∨ ( y ∧ z ) ≈ x ∨ y

( ∨ -semi-distributivity)

Furthermore a lattice L is semi-distributive if is both ∨ -semi-distributive and ∧ -semi-distributive x ∧ y ≈ x ∧ z → x ∧ ( y ∨ z ) ≈ x ∧ y

( ∧ -semi-distributivity) .

N5

M3

1

1

a

0

0

Proof. By assuming x ≤ y , we have x ∨ y = y . Hence, from the distributive property we get:

Definition A.7. Congruence Lattice

An equivalence relation on a set A is a binary relation ∼ that satisfies three properties: reflexivity, symmetry, and transitivity.

Reflexivity: For every element a in A , a is related to itself, denoted as a ∼ a ;

Symmetry: For any elements a and b in A , if a ∼ b , then b ∼ a ;

Transitivity: For any elements a , b , and c in A , if a ∼ b and b ∼ c , then a ∼ c .

In other words, an equivalence relation partitions the set A into subsets, called equivalence classes , such that elements within the same class are equivalent to each other under the relation ∼ .

Let A be an algebra. A congruence θ of A is a equivalent relation on A , that is compatible with the operations of A . Formally, for every n -ary operation f of A : if ( a 1 , b 1 ) , ( a 2 , b 2 ) , . . . , ( a n , b n ) ∈ θ , then ( f ( a 1 , a 2 , . . . , a n ) , f ( b 1 , b 2 , . . . , b n )) ∈ θ . For every algebra A on the set A , the identity relation on A , and A × A are trivial congruences. An algebra with no other congruences is called simple . Let Con( A ) be the set of congruences on the algebra A . Since congruences are closed under intersection, we can define a meet operation: ∧ : Con( A ) × Con( A ) → Con( A ) by simply taking the intersection of the congruences E 1 ∧ E 2 = E 1 ∩ E 2 . Congruences are not closed under union, however we can define the following closure operator of a binary relation E , with respect to a fixed algebra A , such that its image is congruence: ⟨ E ⟩ A = ⋂ { F ∈ Con( A ) | E ⊆ F } . Note that the closure of a binary relation is a congruence and thus depends on the operations in A , not just on the base set. Now define ∨ : Con( A ) × Con( A ) → Con( A ) as E 1 ∨ E 2 = ⟨ E 1 ∪ E 2 ⟩ A . For every algebra A , (Con( A ) , ∧ , ∨ ) with the two operations defined above forms a lattice, called the congruence lattice of A .

A type F is defined as a set of operation symbols along with their respective arities. Each operation symbol represents a specific operation that can be performed on the elements of the algebraic system. To refer to the specific operation performed by a given symbol f on an algebra A of type F , we denote it as f A . This notation allows us to differentiate and access the particular operation carried out by f within the context of A .

Definition A.8. Subalgebra Let A and B be two algebras of the same type. Then B is a subalgebra of A if B ⊆ A and every fundamental operation of B is the restriction of the corresponding operation of A , i.e., for each function symbol f , f B is f A restricted to B .

Definition A.9. Homomorphic image Suppose A and B are two algebras of the same type F , i.e. for each operation of A , there exists a corresponding operation B with the same arity, and vice versa. A mapping α : A → B is called a homomorphism from A to B if

for each n-ary f in F and each sequence a 1 , . . . , a n from A . If, in addition, the mapping α is onto then B is said to be a homomorphic image of A .

Definition A.10. Direct product Let A 1 and A 2 be two algebras of the same type F . We define the direct product A 1 × A 2 to be the algebra whose universe is the set A 1 × A 2 , and such that for f ∈ F and a i ∈ A 1 , a ′ i ∈ A 2 , 1 ≤ i ≤ n ,

The collection of algebraic structures defined by equational laws are called varieties. (15)

Definition A.11. Variety A nonempty class K of algebras of type F is called a variety if it is closed under subalgebras, homomorphic images, and direct products.

B Algorithm 1 details

In the following, we report some technical details on how the dataset generator sketched in Algorithm 1 is actually implemented. First, notice that a brute-force approach is infeasible for large lattices, as given a set of n nodes, the number of binary relations on this set is 2 n 2 . To cope with this issue, first

for each candidate lattice L we consider a squared n × n matrix ≤ L representing its order that has 1 value in a position ( i, j ) if and only if the element i is less or equal to the element j in L (i.e. i ≤ L j ) and 0 otherwise. Then we constraint each matrix to have 1 in the diagonal (reflexivity), 0 in each ( i, j ) with j < i , where ' < ' denotes denotes the order on N (this choice both prunes the majority of isomorphic lattices and yields anti-symmetricity). All the other pairs of elements ( i, j ) can either be such that i ≤ L j or incomparable (i.e. i ≰ L j and j ≰ L i ), and we consider all these possible cases. Finally, we apply matrix multiplications to get a transitive closure of the order relation (convergence guaranteed in at most n -2 steps) and hence ≤ L represents a partial order. To assure that the order represents a lattice, we have to check that each pair of elements ( i, j ) admits a (unique!) infimum and supremum. This step and checking lattice equational properties are implemented tensorially to leverage GPU quicker computations, hence being particularly advantageous when the dimensions of the lattices is such that the computational cost of Algorithm 1 surpasses the overhead of GPU communication. Finally, we notice that even avoiding the isomorphic lattices, for n = 18 there are around 165Bn non-isomorphic different lattices (13). This is why we sampled a fixed number of lattices as n increases, e.g. 20 samples for cardinality after a certain threshold, instead of keeping generating all the possible lattices for each value of n , which is not particularly relevant for our task. Whereas it allows us to study the strong generalization capability of GNNs trained on e.g. up to n = 8 nodes and then evaluated on lattices of higher dimensions, e.g. n = 50 nodes (see Figure 5). Notice that checking e.g. the distributivity for n = 50 is deterministic but requires checking 50 3 identities.

Running time. The running time of the algorithm increases polynomially in the size n of the given lattice (it is O ( n 3 ) for checking each of the equational properties, e.g. distributivity/modularity, and O ( n 4 ) to check if a candidate binary relation is actually a lattice). In a machine with a single quad-core CPU, it requires 20 minutes to generate all the lattices up to N = 8 and sampling n s = 20 lattices for n ∈ [9 , 50] (the dataset we used in the paper).

C Baselines' details

In practice, we train all models using eight message passing layers and different embedding sizes ranging from 16 to 64 . We train all models for 200 epochs with a learning rate of 0 . 001 . For interpretable models, we set the Gumbel-Softmax temperature to the default value of 1 and the activation behavior to "hard," which generates one-hot encoded embeddings in the forward pass, but computes the gradients using the soft scores. For the hierarchical model, we set the internal loss weight to 0 . 1 (to score it roughly 10% less w.r.t. the main loss). Overall, our selection of baselines aims at embracing a wide set of training setups and architectures to assess the effectiveness and versatility of GNNs for analyzing lattice properties in universal algebra. To demonstrate the robustness of our approach, we implemented different types of message passing layers, including graph convolution and GIN.

D Generalization results details

Here we report the raw numbers for the weak and strong generalization results reported as a figure in the main paper. The results are obtained by setting maximum lattice size to 8 in training and using lattices of size 9 or larger during evaluation. All models provide near perfect performance for binary classification and perform slightly worse but still very competitively for multi-label classification. This demonstrates the strong generalization capability of GNNs for universal algebra tasks, and may be an ideal starting point to finding new relevant patterns in UA properties.

In Table 1 we also include a quantitative comparison with GSAT (30), and observe that GSAT and iGNNs obtain comparable results in terms of task generalization and concept completeness (cf. Table 4) when trained on the proposed UA's tasks.

E Concept completeness and purity

Our experimental results (Tables 2 & 4) demonstrate that interpretable GNNs produce concepts with high completeness and low purity, which are standard quantitative metrics used to evaluate the quality of concept-based approaches. Completeness score is the accuracy of a classifier, such as decision

±

±

±

±

±

±

±

±

tree, which takes concepts as inputs and predicts a label. Purity score is the number of graph edits, such as node/edge addition/eliminations, necessary to match two graphs in a cluster. A concept space is said to be pure if the purity score is zero.

We employ decision tree as the classifier, but compute recall instead of accuracy to calculate completeness score since the datasets are heavily unbalanced towards the negative labels. We compute purity scores for each cluster and report the average of those scores as the final purity score. Our approach achieves at least 73% and up to 87% recall, which shows that our interpretable models consistently avoid false negatives in the abundance of negative labels. We obtain around 3-4 purity scores, which suggests that our interpretable models extract relatively pure concept spaces in the presence of large lattices.

Furthermore, the hierarchical structure of interpretable GNNs enables us to evaluate the quality of intermediate concepts layer by layer. This hierarchy provides insights into why we may need more layers, and it can be used as a valuable tool to find the optimal setup and tune the size of the architecture. Additionally, it can also be used to compare the quality of concepts at different layers of the network. To that end, we compare the purity scores of the concept spaces obtained by the second layer and the final layer of HiGNN. As shown in Table 3, deeper layers may produce higher quality concepts for distributivity and join semi-distributivity whereas earlier layers may result in more reliable concepts for the remaining properties. Overall, these results quantitatively assess and validate the high quality of the concepts learned by the interpretable GNNs, highlighting the effectiveness of this approach for learning and analyzing complex algebraic structures.

±

±

±

±

±

±

±

±

±

±

±

±

±

±

±

±

±

±

Graph Concept 0

Concept label:

[0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0]

Graph Concept 1

Concept label:

[0,0,0,0,0,0,0,0,0,0,0,0,1,1,0,0]

Graph Concept 2

Concept label:

[0,0,0,0,0,0,0,0,0,0,0,0,1,1,0,1]

Graph Concept 3

Concept label:

[0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,0]

Graph Concept 4

Concept label:

[0,0,0,0,0,0,0,0,0,0,0,1,1,1,0,0]

Graph Concept 5

Concept label:

[0,0,0,0,0,0,0,0,0,0,1,0,1,1,0,0]

Graph Concept 6

Concept label:

[0,0,0,0,0,0,0,0,0,1,0,0,1,1,0,0]

Graph Concept 7

Concept label:

[0,0,0,0,0,0,0,0,0,1,1,0,0,0,0,1]

Graph Concept 8

Concept label:

[0,0,0,0,0,0,0,0,0,1,1,1,0,0,0,1]

Graph Concept 9

Concept label:

[0,0,0,0,0,0,0,0,1,0,0,0,1,1,0,0]

Graph Concept 10

Concept label:

[0,0,0,0,0,0,0,0,1,1,0,0,0,0,0,1]

Graph Concept 11

Concept label:

[0,0,0,0,0,0,0,0,1,1,1,0,0,0,0,1]

Graph Concept 12

Concept label:

[0,0,0,0,0,0,0,0,1,1,1,1,0,0,0,1]

Graph Concept 13

Concept label:

[0,0,0,0,0,0,0,1,0,0,1,0,0,0,0,1]

Graph Concept 14

Concept label:

[0,0,0,0,0,0,0,1,0,0,1,1,0,0,0,0]

Graph Concept 15

Concept label:

[0,0,0,0,0,0,0,1,0,0,1,1,0,0,0,1]

Graph Concept 16

Concept label:

[0,0,0,0,0,0,0,1,0,1,0,0,0,0,0,1]

Graph Concept 17

Concept label:

[0,0,0,0,0,0,0,1,0,1,0,1,0,0,0,1]

F Concept visualization

Figure 9 visualizes 18 randomly sampled graph concepts (out of the 7896 graph concepts represented by different graph encodings) following the visualization procedure introduced by (28). The figure shows for each concept an example of four (randomly sampled) graphs having the same concept label in the 7-th layer of the hierarchical iGNN trained on the multilabel dataset. Graphs belonging to the same concept show a coherency in their structure and similar patterns. These patterns represent the knowledge extracted and discovered by the hierarchical iGNN.

G Explanations of post-hoc explainers

We compared our Explainable Hierarchical GNN against a standard explainer (namely GNNExplainer (45)) to further support our results. GNNExplainer is the first general, model-agnostic approach for providing interpretable explanations for predictions of any GNN-based model on any graph-based machine learning task and it is widely used in the scientific community as one of the staple explainers in GNN's XAI. In this particular setting, GNNExplainer was configured as follows: model-wise explanation on multiclass-node level classification task, with HiGNN as the model of choice, and GNNExplainer as the desired algorithm, trained for 200 epochs. The explainer takes as input a single graph in the dataset and outputs and explanation for its classification. GNNExplainer will enforce a classification based on the presence or omission of M 3 and/or N 5 and it is possible to visualize the subgraph that lead to this classification by leveraging the visualize_graph function. By doing this, we retrieve the following visualizations:

0

0

1

1

2

7

2

5

3

3

6

4

4

On the right, the substructure identified as N 5 by GNNExplainer which lead to the classification of said graph as non modular and non distributive. On the right, in green M 3 . Our hierarchical model arrives to the same conclusions as the standard explainer but can also be augmented with a standard explainer.

H Code, Licences, Resources

Libraries For our experiments, we implemented all baselines and methods in Python 3.7 and relied upon open-source libraries such as PyTorch 1.11 (33) (BSD license) and Scikit-learn (34) (BSD license). To produce the plots seen in this paper, we made use of Matplotlib 3.5 (BSD license). We will release all of the code required to recreate our experiments in an MIT-licensed public repository.

Resources All of our experiments were run on a private machine with 8 Intel(R) Xeon(R) Gold 5218 CPUs (2.30GHz), 64GB of RAM, and 2 Quadro RTX 8000 Nvidia GPUs. We estimate that approximately 100-GPU hours were required to complete all of our experiments.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the novel application of Artificial Intelligence (AI) in the field of Universal Algebra (UA), which has remained unexplored until now. The authors propose a methodology that combines equational and topological characterizations with graph neural networks to investigate UA conjectures. To address the limitations of transparency and explainability in graph neural networks, they develop an algorithm for generating AI-ready datasets based on UA conjectures and introduce a new interpretable graph network layer. Experimental results demonstrate that interpretable graph networks offer enhanced interpretability without compromising task accuracy, successfully validate existing conjectures, and identify subgraphs that suggest the formulation of new conjectures. Overall, this paper presents a pioneering approach that integrates AI with UA, enabling empirical validation and formulation of conjectures in this foundational field of mathematics.\n\nSTRENGTHS:\nThis paper propose to use GNN to study the UA subject in the modern mathematics. \n\nThe paper claims to have generated the first UA dataset, which is a previously untouched area.\n\nThis paper also proposed the iGNN which improves model interperability and identifies the relevant subgraphs.\n\nWEAKNESSES:\nAs the authors pointed out, the method only applies to finite number of lattices.\n\nQUESTIONS:\nna\n\nLIMITATIONS:\nLimited to finite lattices: The methodology is restricted to finite lattices, which may not capture all the relevant information about infinite algebraic structures. While insights gained from finite-lattice explanations can still provide valuable information, the generalization to infinite structures may be limited.\n\nRestriction to topological properties: The approach focuses on topological properties on graphs, and it may not be applicable to non-structural properties that require the use of other interpretable models. While the method is effective in characterizing equational and topological conjectures, it may not cover a broader range of mathematical properties that are not solely determined by the graph structure.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors explored the use of AI in mathematical problems Universal Algebra. They propose an algorithm for generating datasets of lattice varieties and derive an interpretable graph neural networks for predicting universal algebra’s properties. Experiment results show the effectiveness of the proposed method. Overall, the motivation is clear and the paper is well written.\n\nSTRENGTHS:\nThe problem the paper is investigating is of interest and importance. The authors propose a relatively complete pipeline consisting of both datasets generation and a graph neural network for interpretable prediction.\n\nThe paper is well written and the problem definition is clear. In addition, the authors provide examples to emphasize how the Gumbel softmax works which makes it more readable. \n\nExperiment results seem reasonable. It is good that the authors verify their method on datasets with different settings and use qualitative metrics for interprebility.\n\nWEAKNESSES:\nOne concern is from the dataset generation. As the authors mentioned that the number of existing lattices increases exponentially as n increases, I am curious whether the algorithm is really scalable. In addition, it seems that the algorithm is unable to control label distribution generated.\n\nUsing Gumbel softmax instead of regular activation functions seems novel. However, one concern is whether the semantic concept acquired via the one-hot embedding is really meaningful. Because two embeddings may not necessarily have correlation when they have the maximum value in the same indice. \n\nThe authors mainly compare their method with black-box GNNs and show better results. However, no recent works are included for comparison. Though the authors discussed the difference between the proposed method with existing general graph explanation methods. There are actually some ad-hoc explanation methods that do not rely on subgraph patterns, e.g. GSAT. \n\nRef:\nMiao S, Liu M, Li P. Interpretable and generalizable graph learning via stochastic attention\n\nQUESTIONS:\nWhat is the running time for generating the datasets used in the experiments? How is the time increased with n?\n\nCan the authors include recent general graph explanation methods for comparison?\n\nLIMITATIONS:\nIt is recommended that the authors can provide results in a table to be more readable.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to use AI as a tool to help solve problems in mathematics. In particular, this paper focuses on universal algebra, which studies algebraic structures and can be considered as learning properties of graphs. \n\nThe contributions are two folds:\n- This work proposes a method for generating a benchmark for universal algebra, and releases a dataset of 29000 lattices (partial orderings, which can be represented as graphs) of different sizes and 5 properties.\n  - Generation: filter binary matrices (each lattice on a set of size n can be represented by a nxn matrix) that admits a partial ordering.\n- This work proposes an interpretable neural layer (iGNN) that can be used to solve tasks in the benchmark dataset.\n\nEmpirical results show that the model is able to generalize across lattice sizes while being interpretable.\n\nSTRENGTHS:\n- The paper provides the required background and is self-contained.\n- The proposed benchmark dataset is the first dataset for universal algebra which is valuable to the field.\n- The proposed interpretable layer achieves good empirical performance.\n  - Experiments in Section 5.1 show that on UA tasks, the proposed iGNN scores higher in terms of interpretability metrics, while preserving the performance of black-box GNNs.\n  - The interpretability results in Section 5.2 are interesting: the appearance of substructures corresponding to N5 and M3 is consistent with the theoretical understanding.\n\nWEAKNESSES:\n- The dataset generation process is relatively straightforward (i.e. filtering to get valid lattices, and sampling since it's computationally infeasible to enumerate all configurations).\n  - Update post-rebuttal: this is no longer a concern: the author responses have discussed more details and considerations on scalability. \n- The proposed neural layer is interpretable due to constraining the function classes, which may limit the representational power of the network.\n  - For the pooling layer, the paper chooses max or sum pooling, which reflects the existence or number of 1s in the pooled inputs; this seems to provide only a limited level of interpretability and I wonder if other aspects can be used.\n\nQUESTIONS:\n- Sec 3.2.1, node level features: the \"vice versa\" part is unclear: we only have the converse direction if we assume that the mapping has some reverse Lipschitz condition (e.g. if the neighborhoods are distinct, then the corresponding $h_i$'s are different), which is not guaranteed in the current setup?\n- top of page 5: where do the values of $h_{II}$ come from? Are the values made up as examples?\n- Sec 3.2.3: what are examples of intermediate loss functions? How to obtain the signals/labels for the loss?\n- Sec 5.2: what does a \"concept\" mean? How are the weights computed? Is there a set of \"concepts\" (small graphs?) to search over?\n\nWriting related:\n- A notational question: why $\\approx$ rather than =?\n- The background can be defined more clearly in the main text (or please consider adding pointers to the appendix):\n    - explain that $\\vee, \\wedge$ correspond respectively to join (least upper bound) and meet (greatest lower bound).\n    - explain $M_3$ and $N_5$ are named for \"Modular\" and \"Nonmodular\".\n    - line 77: is there a typo?\n    - to help understand the properties, it might be helpful to point out that not modular implies not distributive.\n- Sec 3.1: are $n,L$ the same thing?\n- Fig 6: where's $M_3$?\n\nLIMITATIONS:\nThe paper discusses limitations that the proposed methods are constrained to finite-size graph and only work with structural properties.\n\nThere is no direct societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nA general methodology is introduced that allows a researcher interested in problems regarding lattices, in the mathematical field of Universal Algebra, to investigate any algebraic property whose validity can be verified on a finite lattice.\nThe paper indicates how a dataset can automatically be generated . Then a new GNN variant is devised, adapted from previous works (e.g. [1]), that is interpretable. This model is compared to the performance of a vanilla, black-box GNN, with all the other model parameters kept identical. \n\n[1] Steve Azzolin, Antonio Longa, Pietro Barbiero, Pietro Liò, and Andrea Passerini. Global ex-\nplainability of gnns via logic combination of learned concepts\n\nSTRENGTHS:\nThe paper discusses how a field that has hitherto not seen a large number of AI applications be treated. Two new GNN variants are proposed, called iGNN and HiGNN.\nI also like the fact that the well-known lattices N_5 and M_3 where used throughout the text as examples of how the learning algorithms works (Example 3.1 and 3.2).\n\nWEAKNESSES:\n__I. Various hyperbolizing adjectives are used throughout the text, that do not provide information:__\n\n- line 18: \"Universal Algebra (UA, (6)) is one of the foundational fields of modern mathematics\" The importance of Universal Algebra is somewhat overstated. While it indeed is a foundational field, it is usually not part of a typical university curriculum.\n\n- line 43 onwards: \"First, we propose a novel algorithm that generates a dataset suitable for training AI models based on an UA equational conjecture. Second, we generate and release the first-ever universal algebra’s dataset compatible with AI [...] Our findings\ndemonstrate the potential of our methodology and open the doors of universal algebra to AI.\"\nThe relevance of the contribution is hyperbolized and overstated, for the following reasons:\n1) Indeed, this may be the first application of AI to universal algebra (although experience has shown that in machine learning often new results are introduced, that under a different name were already known previously -- --, so unless an extremely thorough literature review would have been made, I would not feel confident making this assertion). \n2) It is known that AI is very well suitable to a particular type of mathematics, where the involved objects are discrete (see [1]), so it is not surprising that this approach works well in this setting too.\n3) I am not convinced paper does not \"open doors of universal algebra to AI\" (see also the claim repeated on line 408), since the method is not tied to the field of Universal Algebra, but rather to the objects of study. Universal Algebra has strong connections to model theory, for example (see [2], chapter 5, which is a reference that is cited in the paper), and exploring these using machine-learning is definitely not covered under the current approach.\n\nline 90: \"advanced AI methods\" (what exactly makes an AI method \"advanced\" is debatable)\n\nline 396: \"However, as universal algebra is a foundational branch of modern mathematics, any contribution to this field can already have significant implications in various mathematical disciplines.\"\n\nI feel like these hyped-up claims obscure the actual strong points of the paper: A well-executed experiment with a suitable neural model (GNN) (and other strong points outlined in the Strength section). \n\n__II. Other observations__\n\n- while not a weakness, but a curiosity: it is noteworthy that the lattice examples from Figure 2, page 2, are identical to those from the Wikipedia article on lattices, see Pic 10 and Pic 10 [3]. In Wikipedia, it is mentioned that these are the smallest lattices of their type, but the paper does not mention this (tangential) fact.\n\n- in the appendix, a number of definitions from Universal Algebra are taken from well-knowns, e.g., Definition A.3 corresponds ad literam to Definition 9.3 from [2]. I feel it could be emphasized more strongly that many of the definitions are directly copied from existing sources, rather than merely stating \" More formally (6; 8; 16):\", where their reference 6 is my reference [2] below.\nMore precisely, at the beginning of the appendix, it is stated: \"An n-ary operation on A is a function that takes n elements of A and returns a single element from the set. More formally (6; 8; 16):\" which implies that only the first ensuing definition of arity is taken from references rather than all ensuing definitions.\nIt would be helpful, to either explicitly state that all theorems/definitions are taken from books (and ideally, like in Theorem A.5, show from where the theorem comes).\n\n- on table from the appendix is missing (below line 582): \"Note: ADD TABLE WITH CONCEPT PURITY\". This should be fixed, if the paper is accepted, but doesn't make a difference for acceptance/rejection decision.\n\n[1] A.Z. Wagner, Constructions in combinatorics via neural networks, https://arxiv.org/abs/2104.14516\n[2] Burris S., Sankappanavar H.P., A Course in Universal Algebra, Springer Millenium Edition\n[3] https://en.wikipedia.org/wiki/Lattice_(order)\n\nQUESTIONS:\n- Is there a reason you chose to use linear classifiers instead of decision trees, as the interpretable model of choice?\n\n- consider line 150 (\"To address this, we use a hard Gumbel-Softmax activation\") vs. line 219 (\"The choice of the activation and loss functions to train iGNNs depends on the nature of the task at hand and does not affect their interpretability\"). Does that mean that you use different activation functions at different parts of the GNN?\n\nLIMITATIONS:\nThe problem that is solved is quite narrow, though the authors go out of their way to make sure that this is not emphasized.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}